runner: Remove CGO engines, use llama-server exclusively for GGML models (#16031)
* broad lint fixes to sidestep CI scope glitch * runner: Remove CGO engines, use llama-server exclusively for GGML models Remove the vendored GGML and llama.cpp backend, CGO runner, Go model implementations, and sample. llama-server (built from upstream llama.cpp via FetchContent) is now the sole inference engine for GGUF-based models. (Safetensor based models continue to run on the new MLX engine.) This allows us to more rapidly pick up new capabilities and fixes from llama.cpp as they come out. On windows this now requires recent AMD driver versions to support ROCm v7 as llama.cpp currently does not support building against v6. * llama/compat: load Ollama-format GGUFs in llama-server Squashed from upstream/jmorganca/llama-compat on 2026-04-29. Source tip:0c33775d37. Original source commits: -25223160dllama/compat: add in-memory shim so llama-server can load Ollama-format GGUFs -7449b539allm,server: route Ollama-format gemma3 blobs through llama/compat -436f2e2b1llama/compat: make patch-apply idempotent -8c2c9d4c8llama/compat: extend gemma3 handler to cover 1B and 270M blobs -021389f7bllama/compat: shrink clip.cpp injection from 18 lines to 1 -61b367ec2llama/compat: shrink patch to pure call-site hooks (34 -> 20 lines) -36049361cllama/compat: simplify shim (gemma3-tested) -8fa664865llama/compat: add qwen35moe text handler -db0c74530llama/compat: add qwen35moe vision (clip) support -2a388da77llama/compat: split shared infra into a util TU -9a69a17dcllama/compat: document non-public API dependencies -d0f38a915llama/compat: add gpt-oss and lfm2 handlers -086071822llama/compat: add mistral3 text handler (vision TODO) -63bde9ff7llama/compat: add mistral3 vision (clip) support -3a57b89d5llama/compat: apply LLaMA RoPE permute to mistral3 vision Q/K -99cb87439llama/compat: add qwen35, gemma4, deepseek-ocr handlers -2c7850dballama/compat: add nemotron_h_moe handler (latent FFN + MTP skip) -9e3b54225llama/compat: add llama4 text + clip handlers -034fee349llama/compat: add gemma4 clip handler (gemma4v projector) -9945c5a93server: remove dhiltgen/* compat redirect table -5d4539101llama/compat: rewrite gemma4 tokenizer model to BPE -7e0765327llama/compat: add glm-ocr text handler + text-loader load-op hook -f1bd1a25allama/compat: add glm-ocr clip handler (glm4v projector) -4b5cf3420llama/compat: collapse text-loader hook back to one new patch line -eb4ecf4fcllama/compat: extend gemma4 clip handler to gemma4a (audio) -a23a5e76fllama/compat: fix gemma4a per-block norm tensor mapping -cd2dcaff4llama/compat: add embeddinggemma handler -1ce8a6b26llama/compat: add qwen3-vl + qwen2.5-vl handlers -fd98ffa1ellama/compat: add gemma3n + glm4moelite handlers -cc7bdf0bcllama/compat: handle null buft in maybe_load_tensor -0c33775d3llama/compat: disable mmap when load_op transforms text-side tensors * refine implementation * ci: fix windows MLX build * ci: fix windows llama-server build * ci: fix windows rocm build * ci: windows mlx tuning Shorten long-tail on build, and get OllamaSetup.exe back under 2g limit * ci: fix windows dependencies * win: fix dependency gathering * disable openmp * win: arm64 cross-compile build also DRY out CI steps * scheduler improvements * ci: improvements from #15982 * win: favor ninja for faster developer builds * win: fix build * win: fix arm64 cross-compile * win: avoid spaces in compiler path * misc discovery fixes, and bos handling * lint fixes * win: fix arm cross-compile build/CI bugs * llama.cpp update * win: handle multiple CRT dirs * vulkan: add windows iGPU detection * fix creation bugs for patched models, other refactoring work * tune batch size for better performance * ci and lint fixes * fix repeat_last_n bug * build: revamp build for better developer UX * amd, sampler, qwen3next fixes * version bump * fix mlx build * revamp GPU discovery Scanning the output of llama-server is turning out to be too error prone across llama.cpp updates, so this switches to a thin dynamic library load against the bundled GGML libraries so more details can be gathered from the API. * version bump * missing file * ci: fix cache miss on rocm build * refine vulkan dep handling * fix ps reporting bug on full GPU load * improve cmake wiring for customized local builds * version bump * docker build arg cleanup * improve windows exit error logs * fix community gemma4 support and ci flakes * fix mlx unit test * tighten up ps logic to avoid double counting fit log lines * version bump * fix ps view for full gpu layer offload * add MTP wiring for llama-server and create with GGUFs * pick best template by capabilities * version bump * ci: harden apt repos * remove unused cpu core discovery * adjust batch default logic to reduce OOMs * support larger tool calls * fix audio support, template show * qwen35 mtp patch support * flesh out dtypes * rocm deps * version bump * lint fix * block broken gfx1150 on windows * fix qwen3.5 moe mtp tensors in patch * mmproj oom fallback and vulkan on by default * qwen MTP compat fix * version bump * ci: fix WoA cross-compile * ci: workaround ui tool in cross-compile * version bump * win: enable OpenMP for CPU builds * build: improve developer UX * ci: windows path workaround for CPU build * win: fix WoA dependencies * win: fix large offset reads for mmproj patched loads * version bump * fix vulkan dup detection * add OLLAMA_IGPU_ENABLE and largely disable iGPUs by default * opt-in MTP, win large offset, integraton fixes * fix unit test scheduler interaction hang * fix multi-gpu filtering * version bump * review comments * fix thinking level * fix linux rocm ordering and granite 3.3 template * version bump * ci fix - non-shallow MLX checkout * bypass linux sysfs unit test on windows --------- Co-authored-by: jmorganca <jmorganca@gmail.com>
This commit is contained in:
Daniel Hiltgen
+158
-101
@@ -37,116 +37,150 @@ RUN dnf install -y unzip \
|
||||
ENV CMAKE_GENERATOR=Ninja
|
||||
ENV LDFLAGS=-s
|
||||
|
||||
FROM base AS cpu
|
||||
#
|
||||
# GPU toolchain stages — provide compilers for llama-server GPU builds
|
||||
#
|
||||
|
||||
FROM base AS cpu-deps
|
||||
RUN dnf install -y gcc-toolset-11-gcc gcc-toolset-11-gcc-c++
|
||||
ENV PATH=/opt/rh/gcc-toolset-11/root/usr/bin:$PATH
|
||||
COPY CMakeLists.txt CMakePresets.json .
|
||||
COPY ml/backend/ggml/ggml ml/backend/ggml/ggml
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake --preset 'CPU' \
|
||||
&& cmake --build --preset 'CPU' -- -l $(nproc) \
|
||||
&& cmake --install build --component CPU --strip
|
||||
|
||||
FROM base AS cuda-11
|
||||
ARG CUDA11VERSION=11.8
|
||||
RUN dnf install -y cuda-toolkit-${CUDA11VERSION//./-}
|
||||
ENV PATH=/usr/local/cuda-11/bin:$PATH
|
||||
COPY CMakeLists.txt CMakePresets.json .
|
||||
COPY ml/backend/ggml/ggml ml/backend/ggml/ggml
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake --preset 'CUDA 11' \
|
||||
&& cmake --build --preset 'CUDA 11' -- -l $(nproc) \
|
||||
&& cmake --install build --component CUDA --strip
|
||||
|
||||
FROM base AS cuda-12
|
||||
FROM base AS cuda-12-deps
|
||||
ARG CUDA12VERSION=12.8
|
||||
RUN dnf install -y cuda-toolkit-${CUDA12VERSION//./-}
|
||||
ENV PATH=/usr/local/cuda-12/bin:$PATH
|
||||
COPY CMakeLists.txt CMakePresets.json .
|
||||
COPY ml/backend/ggml/ggml ml/backend/ggml/ggml
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake --preset 'CUDA 12' \
|
||||
&& cmake --build --preset 'CUDA 12' -- -l $(nproc) \
|
||||
&& cmake --install build --component CUDA --strip
|
||||
|
||||
|
||||
FROM base AS cuda-13
|
||||
FROM base AS cuda-13-deps
|
||||
ARG CUDA13VERSION=13.0
|
||||
RUN dnf install -y cuda-toolkit-${CUDA13VERSION//./-}
|
||||
ENV PATH=/usr/local/cuda-13/bin:$PATH
|
||||
COPY CMakeLists.txt CMakePresets.json .
|
||||
COPY ml/backend/ggml/ggml ml/backend/ggml/ggml
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake --preset 'CUDA 13' \
|
||||
&& cmake --build --preset 'CUDA 13' -- -l $(nproc) \
|
||||
&& cmake --install build --component CUDA --strip
|
||||
|
||||
FROM base AS rocm-7-deps
|
||||
ENV PATH=/opt/rocm/llvm/bin:/opt/rocm/hcc/bin:/opt/rocm/hip/bin:/opt/rocm/bin:$PATH
|
||||
|
||||
FROM base AS rocm-7
|
||||
ENV PATH=/opt/rocm/hcc/bin:/opt/rocm/hip/bin:/opt/rocm/bin:/opt/rocm/hcc/bin:$PATH
|
||||
COPY CMakeLists.txt CMakePresets.json .
|
||||
COPY ml/backend/ggml/ggml ml/backend/ggml/ggml
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake --preset 'ROCm 7' \
|
||||
&& cmake --build --preset 'ROCm 7' -- -l $(nproc) \
|
||||
&& cmake --install build --component HIP --strip
|
||||
RUN rm -f dist/lib/ollama/rocm/rocblas/library/*gfx90[06]*
|
||||
|
||||
FROM --platform=linux/arm64 nvcr.io/nvidia/l4t-jetpack:${JETPACK5VERSION} AS jetpack-5
|
||||
ARG CMAKEVERSION
|
||||
ARG NINJAVERSION
|
||||
RUN apt-get update && apt-get install -y curl ccache unzip \
|
||||
&& curl -fsSL https://github.com/Kitware/CMake/releases/download/v${CMAKEVERSION}/cmake-${CMAKEVERSION}-linux-$(uname -m).tar.gz | tar xz -C /usr/local --strip-components 1 \
|
||||
&& curl -fsSL -o /tmp/ninja.zip https://github.com/ninja-build/ninja/releases/download/v${NINJAVERSION}/ninja-linux-aarch64.zip \
|
||||
&& unzip /tmp/ninja.zip -d /usr/local/bin \
|
||||
&& rm /tmp/ninja.zip
|
||||
ENV CMAKE_GENERATOR=Ninja
|
||||
COPY CMakeLists.txt CMakePresets.json .
|
||||
COPY ml/backend/ggml/ggml ml/backend/ggml/ggml
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake --preset 'JetPack 5' \
|
||||
&& cmake --build --preset 'JetPack 5' -- -l $(nproc) \
|
||||
&& cmake --install build --component CUDA --strip
|
||||
|
||||
FROM --platform=linux/arm64 nvcr.io/nvidia/l4t-jetpack:${JETPACK6VERSION} AS jetpack-6
|
||||
ARG CMAKEVERSION
|
||||
ARG NINJAVERSION
|
||||
RUN apt-get update && apt-get install -y curl ccache unzip \
|
||||
&& curl -fsSL https://github.com/Kitware/CMake/releases/download/v${CMAKEVERSION}/cmake-${CMAKEVERSION}-linux-$(uname -m).tar.gz | tar xz -C /usr/local --strip-components 1 \
|
||||
&& curl -fsSL -o /tmp/ninja.zip https://github.com/ninja-build/ninja/releases/download/v${NINJAVERSION}/ninja-linux-aarch64.zip \
|
||||
&& unzip /tmp/ninja.zip -d /usr/local/bin \
|
||||
&& rm /tmp/ninja.zip
|
||||
ENV CMAKE_GENERATOR=Ninja
|
||||
COPY CMakeLists.txt CMakePresets.json .
|
||||
COPY ml/backend/ggml/ggml ml/backend/ggml/ggml
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake --preset 'JetPack 6' \
|
||||
&& cmake --build --preset 'JetPack 6' -- -l $(nproc) \
|
||||
&& cmake --install build --component CUDA --strip
|
||||
|
||||
FROM base AS vulkan
|
||||
FROM base AS vulkan-deps
|
||||
ARG VULKANVERSION
|
||||
RUN ln -s /usr/bin/python3 /usr/bin/python \
|
||||
&& wget https://sdk.lunarg.com/sdk/download/${VULKANVERSION}/linux/vulkansdk-linux-x86_64-${VULKANVERSION}.tar.xz -O /tmp/vulkansdk.tar.xz \
|
||||
&& tar xvf /tmp/vulkansdk.tar.xz -C /tmp \
|
||||
&& /tmp/${VULKANVERSION}/vulkansdk -j 8 vulkan-headers \
|
||||
&& /tmp/${VULKANVERSION}/vulkansdk -j 8 spirv-headers \
|
||||
&& /tmp/${VULKANVERSION}/vulkansdk -j 8 shaderc \
|
||||
&& cp -r /tmp/${VULKANVERSION}/x86_64/include/* /usr/local/include/ \
|
||||
&& cp -r /tmp/${VULKANVERSION}/x86_64/lib/* /usr/local/lib \
|
||||
&& cp -r /tmp/${VULKANVERSION}/x86_64/share/* /usr/local/share/ \
|
||||
&& cp -r /tmp/${VULKANVERSION}/x86_64/bin/* /usr/local/bin/ \
|
||||
&& rm -rf /tmp/${VULKANVERSION} /tmp/vulkansdk.tar.xz
|
||||
COPY CMakeLists.txt CMakePresets.json .
|
||||
COPY ml/backend/ggml/ggml ml/backend/ggml/ggml
|
||||
ENV VULKAN_SDK=/usr/local
|
||||
|
||||
#
|
||||
# llama-server stages — rebuild when LLAMA_CPP_VERSION, llama/server/, or llama/compat/ changes.
|
||||
#
|
||||
# CPU stage: llama-server + ggml-base + ggml-cpu variants → lib/ollama/
|
||||
# GPU stages: GPU backend .so only → lib/ollama/<variant>/
|
||||
#
|
||||
|
||||
FROM cpu-deps AS llama-server-cpu
|
||||
COPY LLAMA_CPP_VERSION .
|
||||
COPY llama/server llama/server
|
||||
COPY llama/compat llama/compat
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake --preset 'Vulkan' \
|
||||
&& cmake --build --preset 'Vulkan' -- -l $(nproc) \
|
||||
&& cmake --install build --component Vulkan --strip
|
||||
cmake -S llama/server --preset cpu \
|
||||
&& cmake --build build/llama-server-cpu -- -l $(nproc) \
|
||||
&& cmake --install build/llama-server-cpu --component llama-server --strip \
|
||||
&& for lib in \
|
||||
/usr/lib64/libgomp.so* \
|
||||
/usr/lib64/libomp.so* \
|
||||
/opt/rh/gcc-toolset-11/root/usr/lib64/libgomp.so* \
|
||||
/opt/rh/gcc-toolset-11/root/usr/lib64/libomp.so*; do \
|
||||
[ -e "$lib" ] && cp -a "$lib" dist/lib/ollama/ || true; \
|
||||
done
|
||||
|
||||
FROM cuda-12-deps AS llama-server-cuda_v12
|
||||
COPY LLAMA_CPP_VERSION .
|
||||
COPY llama/server llama/server
|
||||
COPY llama/compat llama/compat
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake -S llama/server --preset llama_cuda_v12_linux \
|
||||
&& cmake --build build/llama-server-cuda_v12 -- -l $(nproc) \
|
||||
&& cmake --install build/llama-server-cuda_v12 --component llama-server --strip
|
||||
|
||||
FROM cuda-13-deps AS llama-server-cuda_v13
|
||||
COPY LLAMA_CPP_VERSION .
|
||||
COPY llama/server llama/server
|
||||
COPY llama/compat llama/compat
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake -S llama/server --preset llama_cuda_v13_linux \
|
||||
&& cmake --build build/llama-server-cuda_v13 -- -l $(nproc) \
|
||||
&& cmake --install build/llama-server-cuda_v13 --component llama-server --strip
|
||||
|
||||
FROM rocm-7-deps AS llama-server-rocm_v7_2
|
||||
ENV CC=clang CXX=clang++
|
||||
COPY LLAMA_CPP_VERSION .
|
||||
COPY llama/server llama/server
|
||||
COPY llama/compat llama/compat
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake -S llama/server --preset rocm_v7_2_linux \
|
||||
&& cmake --build build/llama-server-rocm_v7_2 -- -l $(nproc) \
|
||||
&& cmake --install build/llama-server-rocm_v7_2 --component llama-server --strip
|
||||
RUN rm -f dist/lib/ollama/rocm_v7_2/rocblas/library/*gfx90[06]*
|
||||
|
||||
FROM vulkan-deps AS llama-server-vulkan
|
||||
COPY LLAMA_CPP_VERSION .
|
||||
COPY llama/server llama/server
|
||||
COPY llama/compat llama/compat
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake -S llama/server --preset vulkan \
|
||||
&& cmake --build build/llama-server-vulkan -- -l $(nproc) \
|
||||
&& cmake --install build/llama-server-vulkan --component llama-server --strip
|
||||
|
||||
#
|
||||
# JetPack stages — self-contained with their own base images
|
||||
#
|
||||
|
||||
FROM --platform=linux/arm64 nvcr.io/nvidia/l4t-jetpack:${JETPACK5VERSION} AS jetpack-5
|
||||
ARG CMAKEVERSION
|
||||
ARG NINJAVERSION
|
||||
RUN apt-get update && apt-get install -y curl ccache git unzip \
|
||||
&& curl -fsSL https://github.com/Kitware/CMake/releases/download/v${CMAKEVERSION}/cmake-${CMAKEVERSION}-linux-$(uname -m).tar.gz | tar xz -C /usr/local --strip-components 1 \
|
||||
&& curl -fsSL -o /tmp/ninja.zip https://github.com/ninja-build/ninja/releases/download/v${NINJAVERSION}/ninja-linux-aarch64.zip \
|
||||
&& unzip /tmp/ninja.zip -d /usr/local/bin \
|
||||
&& rm /tmp/ninja.zip
|
||||
ENV CMAKE_GENERATOR=Ninja
|
||||
COPY LLAMA_CPP_VERSION .
|
||||
COPY llama/server llama/server
|
||||
COPY llama/compat llama/compat
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake -S llama/server --preset llama_cuda_jetpack5 \
|
||||
&& cmake --build build/llama-server-cuda_jetpack5 -- -l $(nproc) \
|
||||
&& cmake --install build/llama-server-cuda_jetpack5 --component llama-server --strip
|
||||
|
||||
FROM --platform=linux/arm64 nvcr.io/nvidia/l4t-jetpack:${JETPACK6VERSION} AS jetpack-6
|
||||
ARG CMAKEVERSION
|
||||
ARG NINJAVERSION
|
||||
RUN apt-get update && apt-get install -y curl ccache git unzip \
|
||||
&& curl -fsSL https://github.com/Kitware/CMake/releases/download/v${CMAKEVERSION}/cmake-${CMAKEVERSION}-linux-$(uname -m).tar.gz | tar xz -C /usr/local --strip-components 1 \
|
||||
&& curl -fsSL -o /tmp/ninja.zip https://github.com/ninja-build/ninja/releases/download/v${NINJAVERSION}/ninja-linux-aarch64.zip \
|
||||
&& unzip /tmp/ninja.zip -d /usr/local/bin \
|
||||
&& rm /tmp/ninja.zip
|
||||
ENV CMAKE_GENERATOR=Ninja
|
||||
COPY LLAMA_CPP_VERSION .
|
||||
COPY llama/server llama/server
|
||||
COPY llama/compat llama/compat
|
||||
RUN --mount=type=cache,target=/root/.ccache \
|
||||
cmake -S llama/server --preset llama_cuda_jetpack6 \
|
||||
&& cmake --build build/llama-server-cuda_jetpack6 -- -l $(nproc) \
|
||||
&& cmake --install build/llama-server-cuda_jetpack6 --component llama-server --strip
|
||||
|
||||
#
|
||||
# MLX stage
|
||||
#
|
||||
|
||||
FROM base AS mlx
|
||||
ARG CUDA13VERSION=13.0
|
||||
# OLLAMA_MLX_BUILD_JOBS empty -> ninja gates by load average (-l $(nproc))
|
||||
ARG OLLAMA_MLX_BUILD_JOBS=
|
||||
ARG OLLAMA_MLX_NVCC_THREADS=2
|
||||
ARG MLX_CUDA_RAM_MB=
|
||||
RUN dnf install -y cuda-toolkit-${CUDA13VERSION//./-} \
|
||||
&& dnf install -y openblas-devel lapack-devel \
|
||||
&& dnf install -y libcudnn9-cuda-13 libcudnn9-devel-cuda-13 \
|
||||
@@ -157,7 +191,7 @@ ENV LAPACK_INCLUDE_DIRS=/usr/include/openblas
|
||||
ENV CGO_LDFLAGS="-L/usr/local/cuda-13/lib64 -L/usr/local/cuda-13/targets/x86_64-linux/lib/stubs"
|
||||
WORKDIR /go/src/github.com/ollama/ollama
|
||||
COPY CMakeLists.txt CMakePresets.json .
|
||||
COPY ml/backend/ggml/ggml ml/backend/ggml/ggml
|
||||
COPY cmake cmake
|
||||
COPY x/imagegen/mlx x/imagegen/mlx
|
||||
COPY go.mod go.sum .
|
||||
COPY MLX_VERSION MLX_C_VERSION .
|
||||
@@ -173,10 +207,12 @@ RUN --mount=type=cache,target=/root/.ccache \
|
||||
&& if [ -f /tmp/local-mlx-c/CMakeLists.txt ]; then \
|
||||
export OLLAMA_MLX_C_SOURCE=/tmp/local-mlx-c; \
|
||||
fi \
|
||||
&& cmake --preset 'MLX CUDA 13' -DBLAS_INCLUDE_DIRS=/usr/include/openblas -DLAPACK_INCLUDE_DIRS=/usr/include/openblas -DCMAKE_CUDA_FLAGS="-t ${OLLAMA_MLX_NVCC_THREADS}" \
|
||||
&& cmake --build --preset 'MLX CUDA 13' -- -l $(nproc) ${OLLAMA_MLX_BUILD_JOBS:+-j ${OLLAMA_MLX_BUILD_JOBS}} \
|
||||
&& cmake --install build --component MLX --strip \
|
||||
&& cmake --install build --component MLX_VENDOR
|
||||
&& cmake -S . -B build/mlx_cuda_v13 -DOLLAMA_MLX_BACKENDS=cuda_v13 -DBLAS_INCLUDE_DIRS=/usr/include/openblas -DLAPACK_INCLUDE_DIRS=/usr/include/openblas -DCMAKE_CUDA_FLAGS="-t ${OLLAMA_MLX_NVCC_THREADS}" ${MLX_CUDA_RAM_MB:+-DMLX_CUDA_RAM_MB=${MLX_CUDA_RAM_MB}} -DOLLAMA_PAYLOAD_INSTALL_PREFIX=/go/src/github.com/ollama/ollama/dist \
|
||||
&& cmake --build build/mlx_cuda_v13 --target ollama-mlx-cuda_v13 -- -l $(nproc) ${OLLAMA_MLX_BUILD_JOBS:+-j ${OLLAMA_MLX_BUILD_JOBS}}
|
||||
|
||||
#
|
||||
# Go build
|
||||
#
|
||||
|
||||
FROM base AS build
|
||||
WORKDIR /go/src/github.com/ollama/ollama
|
||||
@@ -194,38 +230,59 @@ ENV CGO_CXXFLAGS="${CGO_CXXFLAGS}"
|
||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||
go build -trimpath -buildmode=pie -o /bin/ollama .
|
||||
|
||||
#
|
||||
# Assembly stages — combine llama-server variants + GPU runtime libs
|
||||
#
|
||||
|
||||
FROM --platform=linux/amd64 scratch AS amd64
|
||||
# COPY --from=cuda-11 dist/lib/ollama/ /lib/ollama/
|
||||
COPY --from=cuda-12 dist/lib/ollama /lib/ollama/
|
||||
COPY --from=cuda-13 dist/lib/ollama /lib/ollama/
|
||||
COPY --from=vulkan dist/lib/ollama /lib/ollama/
|
||||
COPY --from=llama-server-cpu dist/lib/ollama /lib/ollama/
|
||||
COPY --from=llama-server-cuda_v12 dist/lib/ollama /lib/ollama/
|
||||
COPY --from=llama-server-cuda_v13 dist/lib/ollama /lib/ollama/
|
||||
COPY --from=llama-server-vulkan dist/lib/ollama /lib/ollama/
|
||||
COPY --from=mlx /go/src/github.com/ollama/ollama/dist/lib/ollama /lib/ollama/
|
||||
|
||||
FROM --platform=linux/arm64 scratch AS arm64
|
||||
# COPY --from=cuda-11 dist/lib/ollama/ /lib/ollama/
|
||||
COPY --from=cuda-12 dist/lib/ollama /lib/ollama/
|
||||
COPY --from=cuda-13 dist/lib/ollama/ /lib/ollama/
|
||||
COPY --from=llama-server-cpu dist/lib/ollama /lib/ollama/
|
||||
COPY --from=llama-server-cuda_v12 dist/lib/ollama /lib/ollama/
|
||||
COPY --from=llama-server-cuda_v13 dist/lib/ollama /lib/ollama/
|
||||
COPY --from=jetpack-5 dist/lib/ollama/ /lib/ollama/
|
||||
COPY --from=jetpack-6 dist/lib/ollama/ /lib/ollama/
|
||||
|
||||
FROM scratch AS rocm
|
||||
COPY --from=rocm-7 dist/lib/ollama /lib/ollama
|
||||
COPY --from=llama-server-cpu dist/lib/ollama /lib/ollama
|
||||
COPY --from=llama-server-rocm_v7_2 dist/lib/ollama /lib/ollama
|
||||
|
||||
FROM ${FLAVOR} AS archive
|
||||
COPY --from=cpu dist/lib/ollama /lib/ollama
|
||||
FROM --platform=linux/amd64 scratch AS amd64-archive
|
||||
COPY --from=amd64 /lib/ollama /lib/ollama/
|
||||
COPY --from=llama-server-rocm_v7_2 dist/lib/ollama /lib/ollama/
|
||||
|
||||
FROM --platform=linux/arm64 scratch AS arm64-archive
|
||||
COPY --from=arm64 /lib/ollama /lib/ollama/
|
||||
|
||||
FROM ${TARGETARCH}-archive AS archive
|
||||
COPY --from=build /bin/ollama /bin/ollama
|
||||
|
||||
FROM ${FLAVOR} AS image-archive
|
||||
COPY --from=build /bin/ollama /bin/ollama
|
||||
|
||||
FROM ubuntu:24.04
|
||||
ARG APT_MIRROR=http://archive.ubuntu.com/ubuntu
|
||||
RUN sed -i "s|http://archive.ubuntu.com/ubuntu|$APT_MIRROR|g" /etc/apt/sources.list.d/ubuntu.sources \
|
||||
ARG APT_PORTS_MIRROR=http://ports.ubuntu.com/ubuntu-ports
|
||||
RUN sed -i \
|
||||
-e "s|http://archive.ubuntu.com/ubuntu|$APT_MIRROR|g" \
|
||||
-e "s|http://ports.ubuntu.com/ubuntu-ports|$APT_PORTS_MIRROR|g" \
|
||||
/etc/apt/sources.list.d/ubuntu.sources \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y ca-certificates libvulkan1 libopenblas0 \
|
||||
&& sed -i "s|$APT_MIRROR|http://archive.ubuntu.com/ubuntu|g" /etc/apt/sources.list.d/ubuntu.sources \
|
||||
&& sed -i \
|
||||
-e "s|$APT_MIRROR|http://archive.ubuntu.com/ubuntu|g" \
|
||||
-e "s|$APT_PORTS_MIRROR|http://ports.ubuntu.com/ubuntu-ports|g" \
|
||||
/etc/apt/sources.list.d/ubuntu.sources \
|
||||
&& apt-get clean \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
COPY --from=archive /bin /usr/bin
|
||||
COPY --from=image-archive /bin /usr/bin
|
||||
ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
COPY --from=archive /lib/ollama /usr/lib/ollama
|
||||
COPY --from=image-archive /lib/ollama /usr/lib/ollama
|
||||
ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
|
||||
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
||||
ENV NVIDIA_VISIBLE_DEVICES=all
|
||||
|
||||
Reference in New Issue
Block a user