39 Commits

Author	SHA1	Message	Date
Daniel Hiltgen	9db4bdbad6	runner: Remove CGO engines, use llama-server exclusively for GGML models (#16031) ... * broad lint fixes to sidestep CI scope glitch * runner: Remove CGO engines, use llama-server exclusively for GGML models Remove the vendored GGML and llama.cpp backend, CGO runner, Go model implementations, and sample. llama-server (built from upstream llama.cpp via FetchContent) is now the sole inference engine for GGUF-based models. (Safetensor based models continue to run on the new MLX engine.) This allows us to more rapidly pick up new capabilities and fixes from llama.cpp as they come out. On windows this now requires recent AMD driver versions to support ROCm v7 as llama.cpp currently does not support building against v6. * llama/compat: load Ollama-format GGUFs in llama-server Squashed from upstream/jmorganca/llama-compat on 2026-04-29. Source tip: 0c33775d37. Original source commits: - 25223160d llama/compat: add in-memory shim so llama-server can load Ollama-format GGUFs - 7449b539a llm,server: route Ollama-format gemma3 blobs through llama/compat - 436f2e2b1 llama/compat: make patch-apply idempotent - 8c2c9d4c8 llama/compat: extend gemma3 handler to cover 1B and 270M blobs - 021389f7b llama/compat: shrink clip.cpp injection from 18 lines to 1 - 61b367ec2 llama/compat: shrink patch to pure call-site hooks (34 -> 20 lines) - 36049361c llama/compat: simplify shim (gemma3-tested) - 8fa664865 llama/compat: add qwen35moe text handler - db0c74530 llama/compat: add qwen35moe vision (clip) support - 2a388da77 llama/compat: split shared infra into a util TU - 9a69a17dc llama/compat: document non-public API dependencies - d0f38a915 llama/compat: add gpt-oss and lfm2 handlers - 086071822 llama/compat: add mistral3 text handler (vision TODO) - 63bde9ff7 llama/compat: add mistral3 vision (clip) support - 3a57b89d5 llama/compat: apply LLaMA RoPE permute to mistral3 vision Q/K - 99cb87439 llama/compat: add qwen35, gemma4, deepseek-ocr handlers - 2c7850dba llama/compat: add nemotron_h_moe handler (latent FFN + MTP skip) - 9e3b54225 llama/compat: add llama4 text + clip handlers - 034fee349 llama/compat: add gemma4 clip handler (gemma4v projector) - 9945c5a93 server: remove dhiltgen/* compat redirect table - 5d4539101 llama/compat: rewrite gemma4 tokenizer model to BPE - 7e0765327 llama/compat: add glm-ocr text handler + text-loader load-op hook - f1bd1a25a llama/compat: add glm-ocr clip handler (glm4v projector) - 4b5cf3420 llama/compat: collapse text-loader hook back to one new patch line - eb4ecf4fc llama/compat: extend gemma4 clip handler to gemma4a (audio) - a23a5e76f llama/compat: fix gemma4a per-block norm tensor mapping - cd2dcaff4 llama/compat: add embeddinggemma handler - 1ce8a6b26 llama/compat: add qwen3-vl + qwen2.5-vl handlers - fd98ffa1e llama/compat: add gemma3n + glm4moelite handlers - cc7bdf0bc llama/compat: handle null buft in maybe_load_tensor - 0c33775d3 llama/compat: disable mmap when load_op transforms text-side tensors * refine implementation * ci: fix windows MLX build * ci: fix windows llama-server build * ci: fix windows rocm build * ci: windows mlx tuning Shorten long-tail on build, and get OllamaSetup.exe back under 2g limit * ci: fix windows dependencies * win: fix dependency gathering * disable openmp * win: arm64 cross-compile build also DRY out CI steps * scheduler improvements * ci: improvements from #15982 * win: favor ninja for faster developer builds * win: fix build * win: fix arm64 cross-compile * win: avoid spaces in compiler path * misc discovery fixes, and bos handling * lint fixes * win: fix arm cross-compile build/CI bugs * llama.cpp update * win: handle multiple CRT dirs * vulkan: add windows iGPU detection * fix creation bugs for patched models, other refactoring work * tune batch size for better performance * ci and lint fixes * fix repeat_last_n bug * build: revamp build for better developer UX * amd, sampler, qwen3next fixes * version bump * fix mlx build * revamp GPU discovery Scanning the output of llama-server is turning out to be too error prone across llama.cpp updates, so this switches to a thin dynamic library load against the bundled GGML libraries so more details can be gathered from the API. * version bump * missing file * ci: fix cache miss on rocm build * refine vulkan dep handling * fix ps reporting bug on full GPU load * improve cmake wiring for customized local builds * version bump * docker build arg cleanup * improve windows exit error logs * fix community gemma4 support and ci flakes * fix mlx unit test * tighten up ps logic to avoid double counting fit log lines * version bump * fix ps view for full gpu layer offload * add MTP wiring for llama-server and create with GGUFs * pick best template by capabilities * version bump * ci: harden apt repos * remove unused cpu core discovery * adjust batch default logic to reduce OOMs * support larger tool calls * fix audio support, template show * qwen35 mtp patch support * flesh out dtypes * rocm deps * version bump * lint fix * block broken gfx1150 on windows * fix qwen3.5 moe mtp tensors in patch * mmproj oom fallback and vulkan on by default * qwen MTP compat fix * version bump * ci: fix WoA cross-compile * ci: workaround ui tool in cross-compile * version bump * win: enable OpenMP for CPU builds * build: improve developer UX * ci: windows path workaround for CPU build * win: fix WoA dependencies * win: fix large offset reads for mmproj patched loads * version bump * fix vulkan dup detection * add OLLAMA_IGPU_ENABLE and largely disable iGPUs by default * opt-in MTP, win large offset, integraton fixes * fix unit test scheduler interaction hang * fix multi-gpu filtering * version bump * review comments * fix thinking level * fix linux rocm ordering and granite 3.3 template * version bump * ci fix - non-shallow MLX checkout * bypass linux sysfs unit test on windows --------- Co-authored-by: jmorganca <jmorganca@gmail.com>	2026-05-29 13:35:47 -07:00
Daniel Hiltgen	42e6f56c2a	ci: speed up release builds (#15982) ... * ci: speed up release builds This should help speed things up for release. It also will help speed up local developer builds a little. * ci: dedup linux build steps and optimize * review comments	2026-05-15 14:53:15 -07:00
Daniel Hiltgen	534342e7e2	Update MLX and MLX-C with threading fixes (#15845) ... * Update MLX and MLX-C * Run MLX CGO work on a locked OS thread MLX now relies on OS-thread-local execution state for streams, encoders, and caches. Add an mlxthread executor backed by runtime.LockOSThread and route runner initialization, model load, inference, status memory reads, and cleanup through the worker so Go goroutine migration cannot split MLX state across native threads. Also stop caching default MLX streams before the runner owns the thread and add worker/threaded MLX regression tests. * mlx: use common status writer * mlx: bundle missing libjaccl on arm64 Inspired by #15793 * review comments	2026-05-03 10:03:14 -07:00
Daniel Hiltgen	aec2fef95d	ci: harden cuda include path handling (#15093) ... On windows we can get multiple include dirs, so find where the headers are then copy from that location.	2026-03-27 07:57:07 -07:00
Daniel Hiltgen	516ebd8548	ci: include mlx jit headers on linux (#15083) ... * ci: include mlx jit headers on linux * handle CUDA JIT headers	2026-03-26 23:10:07 -07:00
Daniel Hiltgen	10e51c5177	MLX: add header vendoring and remove go build tag (#14642) ... * prefer rocm v6 on windows Avoid building with v7 - more changes are needed * MLX: add header vendoring and remove go build tag This switches to using a vendoring approach for the mlx-c headers so that Go can build without requiring a cmake first. This enables building the new MLX based code by default. Every time cmake runs, the headers are refreshed, so we can easily keep them in sync when we bump mlx versions. Basic Windows and Linux support are verified. * ci: harden for flaky choco repo servers CI sometimes fails due to choco not actually installing cache. Since it just speeds up the build, we can proceed without. * review comments	2026-03-09 17:24:45 -07:00
Daniel Hiltgen	ebb1b9ec14	rocm: update linux to v7.2 (#14391) ... * rocm: update linux to v7.2 * review comments	2026-03-09 08:26:55 -07:00
Michael Yang	814630ca60	Revert "move tokenizers to separate package (#13825)" (#14111)	2026-02-05 20:49:08 -08:00
Daniel Hiltgen	02a2401596	mlx: bundle openblas dependency (#13706)	2026-01-13 15:29:47 -08:00
Jeffrey Morgan	dbf47ee55a	cmake: use CMAKE_SYSTEM_PROCESSOR instead of CMAKE_OSX_ARCHITECTURES for mlx.metallib install (#13696) ... The CMake condition for installing mlx.metallib checks CMAKE_OSX_ARCHITECTURES, but this variable is only set when explicitly passed - not auto-detected. The arm64 build was missing this flag, causing the metallib to not be installed, which then caused codesign to fail on the unexpanded glob pattern.	2026-01-12 20:05:11 -08:00
Jeffrey Morgan	af7ea6e96e	x/imagegen: install mlx.metallib and fix macOS rpath handling, add mlx library directories to LD_LIBRARY_PATH (#13695) ... - Install mlx.metallib for arm64 builds (required for Metal GPU acceleration) - Apply rpath settings to all macOS builds, not just x86_64 - Add CMAKE_BUILD_WITH_INSTALL_RPATH to avoid install_name_tool errors - Update build_darwin.sh to copy, sign, and package the metallib	2026-01-12 19:03:11 -08:00
Daniel Hiltgen	33ee7168ba	Add experimental MLX backend and engine with imagegen support (#13648) ... * WIP - MLX backend with gemma3 * MLX: add cmake and go tag build toggles To build the new MLX backend code: cmake --preset MLX cmake --build --preset MLX --parallel cmake --install build --component MLX go build -tags mlx . Note: the main.go entrypoint for the MLX engine will change in a follow up commit. * add experimental image generation runtime * add experimental image generation runtime * MLX: wire up cuda build for linux * MLX: get dependencies correct and dedup This is still too large for a unified github artifact, but is now "correct" for the mlx_cuda_v13 directory. * fix relative link bug in dedup * Add darwin build and readme * add go build tag for mlx dependent code and wire up build_darwin.sh * lint cleanup * macos: build mlx for x86 This will be CPU only. * cuda build instructions and fix drift from mlx bump * stale comment * Delete agent helper doc * Clean up readme.md * Revise README for tokenizer clarity and details Updated README to clarify tokenizer functionality and removed correctness section. --------- Co-authored-by: jmorganca <jmorganca@gmail.com>	2026-01-08 16:18:59 -08:00
Nhan Nguyen	aacd1cb394	fix: define GGML_VERSION variables for proper SOVERSION expansion (#13469) ... The ggml/src/CMakeLists.txt uses GGML_VERSION_MAJOR for the shared library SOVERSION property, but these variables were not defined when building from ollama's CMakeLists.txt. This caused libggml-base.so to be named with a literal "SOVERSION" suffix (libggml-base.so.SOVERSION) instead of the actual version number (libggml-base.so.0). The fix adds the required GGML_VERSION_* variables before including the ggml subdirectory. Fixes #13436	2025-12-15 14:42:15 -08:00
Thomas Stocker	2aba569a2a	Vulkan based on #9650 (#11835) ... * implement the vulkan C backend * add support in gpu.go * add support in gen_linux.sh * it builds * fix segfault * fix compilation * fix free memory monitor * fix total memory monitor * update gpu.go * fix build * fix check_perfmon len * remove cap_get_bound check * fix vulkan handle releasing * fix build on federa 40 * fix vulkan on windows * making amdgpu work on arm achitecutre with vulkan * add x86_64 lines in VulkanGlobs and capLinuxGlobs * add aarch64 lines in vulkanGlobs and capLinuxGlobs * Fix variable name * Add vulkan build patch from @jmorganca * Sync vendored ggml to add Vulkan support * Updated dockerfile https://github.com/whyvl/ollama-vulkan/issues/7#issuecomment-2660836871 Signed-off-by: Vadim Grinco <vadim@grinco.eu> * Installing rocm library Signed-off-by: Vadim Grinco <vadim@grinco.eu> * This version works well built based on this: https://github.com/whyvl/ollama-vulkan/issues/7#issuecomment-2660836871 Signed-off-by: Vadim Grinco <vadim@grinco.eu> * Applied 00-fix-vulkan-building.patch Work done by McBane87 here: https://github.com/whyvl/ollama-vulkan/issues/7#issuecomment-2660836871 Signed-off-by: Vadim Grinco <vadim@grinco.eu> * Fixed the "detached head" issues Signed-off-by: Vadim Grinco <vadim@grinco.eu> * Merged in the right direction Signed-off-by: Vadim Grinco <vadim@grinco.eu> * Merging the latest stable (#2) * Applied 00-fix-vulkan-building.patch * Implemented vulkan backend based on the work done by whyvl, Dts0, McBane87 and others Tested on AMD Ryzen 7 8845HS w/ Radeon 780M Graphics with ROCm disabled ``` [GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers) [GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers) [GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers) [GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers) [GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers) time=2025-03-11T13:00:40.793Z level=INFO source=gpu.go:199 msg="vulkan: load libvulkan and libcap ok" time=2025-03-11T13:00:40.877Z level=INFO source=gpu.go:421 msg="error looking up vulkan GPU memory" error="device is a CPU" time=2025-03-11T13:00:40.878Z level=WARN source=amd_linux.go:443 msg="amdgpu detected, but no compatible rocm library found. Either install rocm v6, or follow manual install instructions at https://github.com/ollama/ollama/blob/main/docs/linux.md#manual-install" time=2025-03-11T13:00:40.878Z level=WARN source=amd_linux.go:348 msg="unable to verify rocm library: no suitable rocm found, falling back to CPU" time=2025-03-11T13:00:40.879Z level=INFO source=types.go:137 msg="inference compute" id=0 library=vulkan variant="" compute=1.3 driver=1.3 name="AMD Radeon Graphics (RADV GFX1103_R1)" total="15.6 GiB" available="15.6 GiB" ``` ``` # ollama run phi4:14b >>> /set verbose Set 'verbose' mode. >>> how's it going? Hello! I'm here to help you with any questions or tasks you have. How can I assist you today? 😊 total duration: 3.341959745s load duration: 18.165612ms prompt eval count: 15 token(s) prompt eval duration: 475ms prompt eval rate: 31.58 tokens/s eval count: 26 token(s) eval duration: 2.846s eval rate: 9.14 tokens/s >>> ``` * This is no longer needed Signed-off-by: Vadim Grinco <vadim@grinco.eu> * Fixes SIGSEGV: segmentation violation running gemma3 models on ollama 0.6.0 #21 Patch provided by McBane87 on https://github.com/whyvl/ollama-vulkan/issues/21 Signed-off-by: Vadim Grinco <vadim@grinco.eu> * Applied 04-disable-mmap-vulkan.patch From: https://github.com/whyvl/ollama-vulkan/issues/7#issuecomment-2660836871 Signed-off-by: Vadim Grinco <vadim@grinco.eu> * Pulled new upstream code for ggml-bulkan backend Signed-off-by: Vadim Grinco <vadim@grinco.eu> * Merged latest ollama 0.6.2 and nasrally's Flash Attention patches (#5) * readme: add Ellama to list of community integrations (#9800) * readme: add screenpipe to community integrations (#9786) * Add support for ROCm gfx1151 (#9773) * conditionally enable parallel pipelines * sample: make mutations in transforms explicit (#9743) * updated minP to use early exit making use of sorted tokens * ml/backend/ggml: allocate memory with malloc when loading model (#9822) * runner: remove cache prompt flag from ollama runner (#9826) We do not need to bypass the prompt caching in the ollama runner yet, as only embedding models needed to bypass the prompt caching. When embedding models are implemented they can skip initializing this cache completely. * ollamarunner: Check for minBatch of context space when shifting Models can specify that a group of inputs need to be handled a single batch. However, context shifting didn't respect this and could trigger a break anyways. In this case, we should instead trigger a context shift earlier so that it occurs before the grouped batch. Note that there still some corner cases: - A long prompt that exceeds the context window can get truncated in the middle of an image. With the current models, this will result in the model not recognizing the image at all, which is pretty much the expected result with truncation. - The context window is set less than the minimum batch size. The only solution to this is to refuse to load the model with these settings. However, this can never occur with current models and default settings. Since users are unlikely to run into these scenarios, fixing them is left as a follow up. * Applied latest patches from McBane87 See this for details: https://github.com/whyvl/ollama-vulkan/issues/7#issuecomment-2708820861 Signed-off-by: Vadim Grinco <vadim@grinco.eu> * Add ability to enable flash attention on vulkan (#4) * discover: add flash attention handling for vulkan * envconfig: fix typo in config.go As part of the process some code was refactored and I added a new field FlashAttention to GpuInfo since the previous solution didn't allow for a granular check via vulkan extensions. As a side effect, this now allows for granular per-device FA support checking in other places --------- Signed-off-by: Vadim Grinco <vadim@grinco.eu> Co-authored-by: zeo <108888572+zeozeozeo@users.noreply.github.com> Co-authored-by: Louis Beaumont <louis.beaumont@gmail.com> Co-authored-by: Daniel Hiltgen <dhiltgen@users.noreply.github.com> Co-authored-by: Michael Yang <mxyng@pm.me> Co-authored-by: Parth Sareen <parth.sareen@ollama.com> Co-authored-by: Jeffrey Morgan <jmorganca@gmail.com> Co-authored-by: Bruce MacDonald <brucewmacdonald@gmail.com> Co-authored-by: Jesse Gross <jesse@ollama.com> Co-authored-by: Nikita <50599445+nasrally@users.noreply.github.com> * Revert Readme changes * Revert * Revert changes in amd_linux.go * Revert changes in amd_linux.go * Remove flashattention setting gpu.go * Revert whitespace changes in gpu.go * Revert changes in transforms_test.go * Revert changes in runner.go * Revert changes in Makefile.sync * Revert some unintented changes in Dockerfile * Revert vulkan copy changes in Dockerfile * Update Vulkan Code to de4c07f93783a1a96456a44dc16b9db538ee1618 * Fixed duplicate sync in ggml.go * Revert changes in ggml.go * Revert chnages in ggml.go * enable falsh attention on vulkan * revert remove parenthesis * fixed flash attention logic enabling * vk_check_flash_attention 0 means supported * Update gpu.go * Add vulkan to Windows Build script * Remove commented out code * Enable Vulkan Flash attention in FlashAttentionSupported * Fix logging * Update Vulkan backend to e54d41befcc1575f4c898c5ff4ef43970cead75f * Removed libcap related code libcap is not directly related to Vulkan and should be added by its own PR. It adds additional library dependencies for building and also requires users to run setcap or run ollama as root, which is not ideal for easy use * Fix Unit Test (Add Vulkan Library) * Add vulkan to TestHomogeneousGPUs Test * vulkan: get GPU ID (ollama v0.11.5) Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com> * disable mmap for vulkan * Reduce Changes remove TestHomogeneousGPUs (doesn't exist on master) * Update vulkan version to the version used in llama.cpp * rename gpu patch to correct number * added Vulkan API to get correct Device UUID current UUID from pipelineCacheUUID does not match CUDA * Fix GPU ID Patch * Remove Code not in llama.cpp * modified UUID code inside ggml * Fix Patch * Copied minimal definition from vulkan header * Fix compile error in Mac Metal is preferred so we're disabling Vulkan for now * Removed unused code Fix linter error in CI * Fix patches apply * fixing lint error * Removed unneeded function call Somehow removing this call fixed the crashing when Vulkan header was removed * added missing NL * Fixed missing members in Vulkan header also added zero clear for some structs * Fixed wrong structure ID * Fixed Vulkan header More aligned with official header definition now * buildvulkanAsSeperateFunction * Vulkan on Windows Test * temporarly comment out gate to run windows task * use temporarly windows-latest for build * Commenting out other presets to build vulkan * reenable cpu * commenting out error action stop * temporarly commenting out rocm * set vulkan path * comment out cude for faster turnaround * correct vulkan install * correct vulkan silent install * fixed install command * revert debugging changes (vulkan builds on windows) * revert windows-latest * trying to build vulkan for linux * temporarly disable cuda and rocm * try again linux build * fix version * trying to fix * trying again * trying again * fix version * fixed vulkan-sdk name * try again * trying again * try without version number * try again * add some more extra * trying to use version 1.4.313 * revert debugging changes * Filter out already supported gpus * revert debug code * Use runners for GPU discovery This revamps how we discover GPUs in the system by leveraging the Ollama runner. This should eliminate inconsistency between our GPU discovery and the runners capabilities at runtime, particularly for cases where we try to filter out unsupported GPUs. Now the runner does that implicitly based on the actual device list. In some cases free VRAM reporting can be unreliable which can leaad to scheduling mistakes, so this also includes a patch to leverage more reliable VRAM reporting libraries if available. Automatic workarounds have been removed as only one GPU leveraged this, which is now documented. This GPU will soon fall off the support matrix with the next ROCm bump. Additional cleanup of the scheduler and discovery packages can be done in the future once we have switched on the new memory management code, and removed support for the llama runner. * timing info for runner * WIP - wire up Vulkan with the new engine based discovery Not a complete implementation - free VRAM is better, but not accurate on windows * fix - trust the library paths from discovery when starting runner * fix index bug * fix vulkan ids to be underlying * fix - give bootstrapping more time on slow systems * Test if Vulkan device is supported * vk_check_flash_attention is not needed (coompat2 coopmapt and scalar implementation exist) * Handle GGML_VK_VISIBLE_DEVICES * ask for supported first * win: fix CPU query buffer handling Try in a short loop until we get the size right. * test: harden integration tests for slow start If the server takes a while to start up, block tests from starting until it's online to avoid setting large timeouts in individual test cases. * gofumpt fix * fix build * merge fixes * merge fixes * fixed build * merge fixes * fixing build * fixed build * fixed formatting * fixed build * fix vulkan gpu id patch * sync llama.cpp vulkan code * update build windows script * merge fixes * fix format * fixed vulkan casing * handle igpu as gpu * improve case * print out unknown library * rturn Vulkan for vulkan library * Revert "rturn Vulkan for vulkan library" This reverts commit 690461a12fd5e93295d174c97edefb2bc33285b1. * fixed patch number * return Library Name * remvoe debug code * return integrated in vulkan backend * Return pci Properties * update patch * directly get pci proeprties without parsing * workaround for filtering devices. Correct way is to have a LibraryPosition Parameter in the deviceInfo * Revert "directly get pci proeprties without parsing" This reverts commit 8e0624851f5ed7d9f74518f574dfb422e4dd4dc2. * Set FilteredID for Environment Filtering * ROCm Library is named ROCm * revert changes in patch * Create 0028-vulkan-pci-and-memory.patch * vulkan memory patch * casing fix * Add more pci properties * Added better memory management * Added better memory managament * fixed patch * Fixed patch * FilterID creation group by library * filter out vulkan supported by other gpu * fixing deviceid compare * Vulkan Fix FA coopmat1 invalid array indexing * Use everywhere the same Vulkan Version 1.4.321.1 * Remove unneeded patch * vulkan update * sync vulkan glsl files * only use for vulkan the filteredid (numeric device number) * simplify code --------- Signed-off-by: Vadim Grinco <vadim@grinco.eu> Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com> Co-authored-by: pufferffish <github@bandersnatch.anonaddy.com> Co-authored-by: KOISHI KOMEIJI FROM TOUHOU 11 <fuck> Co-authored-by: DSLstandard <qgeneral35@gmail.com> Co-authored-by: pufferffish <me@windtfw.com> Co-authored-by: yeongbba <yeongmo.lee@logpresso.com> Co-authored-by: tomaThomas <tomathomas@mailbox.org> Co-authored-by: Antoine Viallon <antoine@lesviallon.fr> Co-authored-by: Vadim Grinco <vadim@grinco.eu> Co-authored-by: zeo <108888572+zeozeozeo@users.noreply.github.com> Co-authored-by: Louis Beaumont <louis.beaumont@gmail.com> Co-authored-by: Daniel Hiltgen <dhiltgen@users.noreply.github.com> Co-authored-by: Michael Yang <mxyng@pm.me> Co-authored-by: Parth Sareen <parth.sareen@ollama.com> Co-authored-by: Jeffrey Morgan <jmorganca@gmail.com> Co-authored-by: Bruce MacDonald <brucewmacdonald@gmail.com> Co-authored-by: Jesse Gross <jesse@ollama.com> Co-authored-by: Nikita <50599445+nasrally@users.noreply.github.com> Co-authored-by: Masato Nakasaka <masato.nakasaka@intel.com> Co-authored-by: Xiaodong Ye <xiaodong.ye@mthreads.com> Co-authored-by: Daniel Hiltgen <daniel@ollama.com>	2025-10-14 10:59:58 -07:00
Daniel Hiltgen	c68f367ef6	Update GGML to b6646 (#12245) ... Notable EOLs with this change: - MacOS v12 and v13 are no longer supported (v14+ required) - AMD gfx900 and gfx906 are no longer supported	2025-10-02 14:47:10 -07:00
Daniel Hiltgen	bc8909fb38	Use runners for GPU discovery (#12090) ... This revamps how we discover GPUs in the system by leveraging the Ollama runner. This should eliminate inconsistency between our GPU discovery and the runners capabilities at runtime, particularly for cases where we try to filter out unsupported GPUs. Now the runner does that implicitly based on the actual device list. In some cases free VRAM reporting can be unreliable which can leaad to scheduling mistakes, so this also includes a patch to leverage more reliable VRAM reporting libraries if available. Automatic workarounds have been removed as only one GPU leveraged this, which is now documented. This GPU will soon fall off the support matrix with the next ROCm bump. Additional cleanup of the scheduler and discovery packages can be done in the future once we have switched on the new memory management code, and removed support for the llama runner.	2025-10-01 15:12:32 -07:00
Michael Yang	0469861d9d	build: call find_package to instantiate library paths	2025-09-30 13:12:46 -07:00
羊撅撅	c47154c08d	fix: correct condition for AMDGPU_TARGETS filtering logic (#12412)	2025-09-26 11:38:47 -07:00
Daniel Hiltgen	e4ce68311a	cuda: remove compression for better compatibility (#12259) ... This retains compatibility with driver 531 and up at the trade-off of space.	2025-09-12 07:59:14 -07:00
Daniel Hiltgen	17a023f34b	Add v12 + v13 cuda support (#12000) ... * Add support for upcoming NVIDIA Jetsons The latest Jetsons with JetPack 7 are moving to an SBSA compatible model and will not require building a JetPack specific variant. * cuda: bring back dual versions This adds back dual CUDA versions for our releases, with v11 and v13 to cover a broad set of GPUs and driver versions. * win: break up native builds in build_windows.ps1 * v11 build working on windows and linux * switch to cuda v12.8 not JIT * Set CUDA compression to size * enhance manual install linux docs	2025-09-10 12:05:18 -07:00
Michael Yang	1a19df1f3a	update vendored llama.cpp and ggml (#11823) ... * TEMPORARY: Update the llama.cpp upstream to my fork's Granite Four branch This will be redone once my branch is merged upstream in llama.cpp * feat: Update all patches There are a number that are no longer needed at all: - 0003-embeddings: Embeddings entirely overhauled on master - 0008-ensure-KV-cache-is-fully-defragmented: KV caching entirely overhauled on master - 0019-metal-add-mean-kernel-14267: Merged upstream - 0020-CUDA-add-mean-operation-14313: Merged upstream * feat: Sync llama.cpp and ggml * fix: Update rsync-filter for all moved/new/removed files * fix: Add files missing from sync * fix: Update ggml rsync-filter for new ggml-cpu/arch subdirs * fix: Add ggml files missing from sync * fix: Narrow llama.cpp rsync-filter to not include mtmd main tool cpp files * fix: Remove mtmd main cpp files * fix: Add missing include in sampling_ext.cpp * fix: Update llama.go to use mtmd instead of clip/llava * fix: Add patch for mtmd_input_text * chore: Ignore *.patched in the patch directory * fix: Fix support for arch-specific ggml-cpu source files with new arrangement In https://github.com/ggml-org/llama.cpp/pull/13892, all arch-specific implementations were split out into a nested tree structure under ggml-cpu/arch. This conflicts with standard CGO layout where all arch-specific source files are expected to live in the same directory as the parent go module and use suffixes based on GOOS and GOARCH. As such, there were really two options for getting this to work: 1. Add a patch on top of the GGML sync to rearrange the files to match the GO layout convention 2. Use CGO directives to conditionally include the nested source files in the compilation units This commit does (2) in order to minimize the set of changes needed on top of the upstream file layout. To get this to work, there are two key things needed: 1. In cpu.go, #cgo directives are added to explicitly set __${GOARCH}__ in the preprocessor directives 2. In arch-impls.c|cpp, use an #ifdef | #elif defined | #endif chain to explicitly include the .c|.cpp files for the given architecture from the nested directory * fix: Use mtmd_helper to correctly load the bitmap for the image * fix: Apply patch for mtmd_text_input * fix: Add missing stb to llama.cpp rsync-filter * fix: Add sync'ed stb vendored header * fix: Use c++17 and include vendor for go wrapper modules * fix: Update patch 0015 for upstream implementation of uuid * feat: Bump to the latest tip of the branch * fix: Update patches for bump * feat: Bump back to the cenral repo and point at the latest master This includes granite 4 and a number of other model architectures! * fix: Revert changes to ggml export GPU UUID patch * fix: Add patch for GGML_VERSION and GGML_COMMIT constants * feat: Sync all patched code * build: Include cmake/common.cmake in ggml sync * build: Add top-level include for GNUINstallDirs in CMakeLists.txt This is used to populate CMAKE_INSTALL_BINDIR * fix: Add a patch to avoid power throttling API on non-msvc windows builds * fix: Sync patch changes for ggml-cpu.c * feat: Bump llama.cpp to 4a4f42 This picks up support for Kimi K2 and PLaMO-2 * feat: Sync llama.cpp * fix: Handle multi-chunk image encodings from mtmd * fix: Re-number patches after merge with `main` * feat: Bump to 41e78c in the makefile * fix: Fix Solar and argsort/copy patches after bump * fix: Remove Gemma3n CUDA Graphs patch It was implemented upstream: https://github.com/ggml-org/llama.cpp/pull/14741 * feat: Sync llama.cpp / ggml after latest bump * build: Remove unnecessary CFLAGS definitions in cpu.go * fix: Remove unnecessary additions in the rsync-filter * fix: Remove unused vendored code for chat template parsing * Revert "fix: Remove Gemma3n CUDA Graphs patch" This reverts commit d724caced3ce21f08924d4b7801f94ce6638f6ea. * fix: Update 0020 CUDA Graphs for gemma3n to keep both llama.cpp and ollama fixes https://github.com/ollama/ollama/pull/11195#issuecomment-3137312394 * fix: Sync ggml-cuda.cu after keeping both style cuda graph fixes for gemma3n * unwind mxfp4 patch Prepare to bump ggml with their impl for mxfp4 * bump * fix windows build error * Convert tensors at load time Repack the mxfp4 tensors as ggmls kernels expect them to be. * convert mlp bf16 to f32 * buffer the conversion better * reshape earlier * openai swiglu * add ids * split qkv, gate_up * fix nested alt tags * fast attention * remove debug messages * fix lint * remove redundant test * remap values only if source/target are different * add back i32->i32 copy * refactor cpu quants * clean up vendor * update patch instructions * clean up patches * remove webgpu * update mem * also handle gpt-oss * revert convert changes --------- Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Daniel Hiltgen <daniel@ollama.com>	2025-08-14 14:42:58 -07:00
Daniel Hiltgen	1c6669e64c	Re-remove cuda v11 (#10694) ... * Re-remove cuda v11 Revert the revert - drop v11 support requiring drivers newer than Feb 23 This reverts commit c6bcdc4223. * Simplify layout With only one version of the GPU libraries, we can simplify things down somewhat. (Jetsons still require special handling) * distinct sbsa variant for linux arm64 This avoids accidentally trying to load the sbsa cuda libraries on a jetson system which results in crashes. * temporary prevent rocm+cuda mixed loading	2025-06-23 14:07:00 -07:00
Michael Yang	139f84cf21	fix cmakelists (#10804) ... this fixes an issue introduced in #10788	2025-05-21 09:52:52 -07:00
Michael Yang	375839ea2d	chore: disable debug in binary libraries (#10788)	2025-05-21 09:39:38 -07:00
Jeffrey Morgan	09bb2e30f6	ml/backend/ggml: use default CUDA compression mode (#10314)	2025-04-16 19:54:20 -07:00
Jeffrey Morgan	dc264be6ff	ml: add missing cmake property and remove additional CMakeLists.txt (#10310)	2025-04-16 18:56:29 -07:00
Jeffrey Morgan	943464ccb8	llama: update to commit 71e90e88 (#10192)	2025-04-16 15:14:01 -07:00
saman-amd	ead27aa9fe	Add gfx1200 & gfx1201 support on linux (#9878)	2025-03-27 07:35:19 -07:00
Jeffrey Morgan	96a97adf9b	build: use correct GGML_HIP_NO_VMM compiler definition for ggml-hip (#9451)	2025-03-01 17:00:31 -08:00
Jeffrey Morgan	e75c6126e9	build: set GGML_CUDA_NO_VMM for ggml-hip target (#9449)	2025-03-01 14:02:19 -08:00
Michael Yang	b42aba40ed	cuda: enable flash attention ... ggml added an option to disable flash attention so explicitly enable it	2025-02-28 19:40:34 +00:00
Michael Yang	08a299e1d0	cmake: avoid building intel backends on linux	2025-02-18 22:17:00 +00:00
Jeffrey Morgan	a4f69a0191	build: add -DGGML_CUDA_NO_PEER_COPY=ON for rocm builds on windows (#9060)	2025-02-13 00:23:17 -08:00
Michael Yang	abb8dd57f8	add gfx instinct gpus (#8933)	2025-02-07 09:51:22 -08:00
Michael Yang	ae7e368f75	build(rocm): add numa, elf (#8900)	2025-02-06 15:46:30 -08:00
Michael Yang	9a4757ae66	build(rocm): add tinfo (#8899)	2025-02-06 15:08:12 -08:00
Michael Yang	b698f9a0d8	build: add missing dependencies (#8896)	2025-02-06 13:12:16 -08:00
Jeffrey Morgan	4759ecae19	ml/backend/ggml: fix library loading on macOS amd64 (#8827)	2025-02-04 15:05:39 -08:00
Michael Yang	dcfb7a105c	next build (#8539) ... * add build to .dockerignore * test: only build one arch * add build to .gitignore * fix ccache path * filter amdgpu targets * only filter if autodetecting * Don't clobber gpu list for default runner This ensures the GPU specific environment variables are set properly * explicitly set CXX compiler for HIP * Update build_windows.ps1 This isn't complete, but is close. Dependencies are missing, and it only builds the "default" preset. * build: add ollama subdir * add .git to .dockerignore * docs: update development.md * update build_darwin.sh * remove unused scripts * llm: add cwd and build/lib/ollama to library paths * default DYLD_LIBRARY_PATH to LD_LIBRARY_PATH in runner on macOS * add additional cmake output vars for msvc * interim edits to make server detection logic work with dll directories like lib/ollama/cuda_v12 * remove unncessary filepath.Dir, cleanup * add hardware-specific directory to path * use absolute server path * build: linux arm * cmake install targets * remove unused files * ml: visit each library path once * build: skip cpu variants on arm * build: install cpu targets * build: fix workflow * shorter names * fix rocblas install * docs: clean up development.md * consistent build dir removal in development.md * silence -Wimplicit-function-declaration build warnings in ggml-cpu * update readme * update development readme * llm: update library lookup logic now that there is one runner (#8587) * tweak development.md * update docs * add windows cuda/rocm tests --------- Co-authored-by: jmorganca <jmorganca@gmail.com> Co-authored-by: Daniel Hiltgen <daniel@ollama.com>	2025-01-29 15:03:38 -08:00