Mirror_Repos/ollama_ollama
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

llama: fix fattn-tile shared memory overflow on sm_50/52 (#13872)
release / setup-environment (push) Has been cancelled
Details
release / windows-depends (amd64, , https://sdk.lunarg.com/sdk/download/1.4.321.1/windows/vulkansdk-windows-X64-1.4.321.1.exe, windows, Vulkan, vulkan) (push) Has been cancelled
Details
release / windows-depends (amd64, -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_FLAGS="-parallel-jobs=4 -Wno-ignored-attributes -Wno-deprecated-pragma" -DCMAKE_CXX_FLAGS="-parallel-jobs=4 -Wno-ignored-attributes -Wno-deprecated-pragma", https:/… (push) Has been cancelled
Details
release / windows-depends (amd64, ["cudart" "nvcc" "cublas" "cublas_dev" "crt" "nvvm" "nvptxcompiler"], 13.0, , https://developer.download.nvidia.com/compute/cuda/13.0.0/local_installers/cuda_13.0.0_windows.exe, windows, CUDA 13) (push) Has been cancelled
Details
release / windows-depends (amd64, ["cudart" "nvcc" "cublas" "cublas_dev"], 12.8, , https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda_12.8.0_571.96_windows.exe, windows, CUDA 12) (push) Has been cancelled
Details
release / windows-depends (amd64, windows, CPU) (push) Has been cancelled
Details
release / darwin-build (push) Has been cancelled
Details
release / windows-build (amd64, x86_64, windows) (push) Has been cancelled
Details
release / windows-build (arm64, aarch64, windows) (push) Has been cancelled
Details
release / windows-app (push) Has been cancelled
Details
release / linux-build (amd64, linux, archive) (push) Has been cancelled
Details
release / linux-build (amd64, linux, rocm) (push) Has been cancelled
Details
release / linux-build (arm64, linux, archive) (push) Has been cancelled
Details
release / docker-build-push (amd64, CGO_CFLAGS CGO_CXXFLAGS GOFLAGS , linux) (push) Has been cancelled
Details
release / docker-build-push (amd64, CGO_CFLAGS CGO_CXXFLAGS GOFLAGS FLAVOR=rocm , linux, -rocm) (push) Has been cancelled
Details
release / docker-build-push (arm64, CGO_CFLAGS CGO_CXXFLAGS GOFLAGS , linux) (push) Has been cancelled
Details
release / docker-merge-push () (push) Has been cancelled
Details
release / docker-merge-push (-rocm) (push) Has been cancelled
Details
release / release (push) Has been cancelled
Details

Browse Source 
Use nthreads=128 for ncols=4 configurations in flash attention tile
kernel to reduce shared memory usage below 48KB limit on Maxwell
architectures (sm_50/52).

With nthreads=256 and ncols=4, np=2 which caused shared memory to
exceed 48KB. With nthreads=128 and ncols=4, np=1 keeps shared memory
under the limit.
This commit is contained in:
Jeffrey Morgan
2026-01-23 19:22:32 -08:00
committed by GitHub
parent aae6ecbaff
commit 912d984346
2 changed files with 18 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
llama/patches/0032-ggml-enable-MLA-flash-attention-for-GLM-4.7-flash.patch
+11 -12
View File
@@ -17,7 +17,6 @@ CUDA changes:
- Add tile configs for (576, 512, 4) and (576, 512, 8)
- Add MMA config cases for ncols 4
- Add template instances for ncols2=4
- Fix nbatch_fa values in nvidia_fp32 config (32->64)
---
ggml/src/ggml-cuda/fattn-mma-f16.cuh | 15 ++++++++++++---
ggml/src/ggml-cuda/fattn-tile.cuh | 18 +++++++++++++++++-
@@ -83,19 +82,19 @@ index 7c4d6fe67..6389ba5c4 100644
GGML_CUDA_FATTN_TILE_CONFIG_CASE(256, 256, 16, 256, 2, 64, 64)
GGML_CUDA_FATTN_TILE_CONFIG_CASE(256, 256, 32, 256, 2, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 4, 256, 2, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 4, 128, 2, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 8, 256, 2, 64, 64)
GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 16, 256, 2, 64, 64)
return 0;
@@ -122,7 +124,9 @@ static constexpr __host__ __device__ uint32_t ggml_cuda_fattn_tile_get_config_nv
GGML_CUDA_FATTN_TILE_CONFIG_CASE(256, 256, 16, 256, 2, 32, 128)
GGML_CUDA_FATTN_TILE_CONFIG_CASE(256, 256, 32, 256, 2, 32, 64)
- GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 16, 256, 2, 32, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 4, 256, 2, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 8, 256, 2, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 16, 256, 2, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 4, 128, 2, 32, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 8, 256, 2, 32, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 16, 256, 2, 32, 64)
return 0;
}
@@ -103,17 +102,17 @@ index 7c4d6fe67..6389ba5c4 100644
GGML_CUDA_FATTN_TILE_CONFIG_CASE(256, 256, 16, 256, 2, 32, 128)
GGML_CUDA_FATTN_TILE_CONFIG_CASE(256, 256, 32, 256, 2, 32, 128)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 4, 256, 2, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 4, 128, 2, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 8, 256, 2, 64, 64)
GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 16, 256, 2, 64, 64)
GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 32, 512, 1, 128, 64)
@@ -245,6 +251,8 @@ static constexpr __host__ __device__ uint32_t ggml_cuda_fattn_tile_get_config_am
GGML_CUDA_FATTN_TILE_CONFIG_CASE(256, 256, 16, 256, 5, 32, 256)
GGML_CUDA_FATTN_TILE_CONFIG_CASE(256, 256, 32, 256, 3, 64, 128)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 4, 256, 4, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 8, 256, 4, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 4, 128, 2, 64, 64)
+ GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 8, 256, 2, 64, 64)
GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 16, 256, 4, 64, 64)
GGML_CUDA_FATTN_TILE_CONFIG_CASE(576, 512, 32, 256, 2, 128, 64)