Daniel Hiltgen
6286d9a3a5
release / setup-environment (push) Has been cancelled
release / windows-depends (amd64, , https://sdk.lunarg.com/sdk/download/1.4.321.1/windows/vulkansdk-windows-X64-1.4.321.1.exe, windows, Vulkan, vulkan) (push) Has been cancelled
release / windows-depends (amd64, -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_FLAGS="-parallel-jobs=4 -Wno-ignored-attributes -Wno-deprecated-pragma" -DCMAKE_CXX_FLAGS="-parallel-jobs=4 -Wno-ignored-attributes -Wno-deprecated-pragma", https:/… (push) Has been cancelled
release / windows-depends (amd64, ["cudart" "nvcc" "cublas" "cublas_dev" "crt" "nvvm" "nvptxcompiler"], 13.0, , https://developer.download.nvidia.com/compute/cuda/13.0.0/local_installers/cuda_13.0.0_windows.exe, windows, CUDA 13) (push) Has been cancelled
release / windows-depends (amd64, ["cudart" "nvcc" "cublas" "cublas_dev"], 12.8, , https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda_12.8.0_571.96_windows.exe, windows, CUDA 12) (push) Has been cancelled
release / windows-depends (amd64, windows, CPU) (push) Has been cancelled
release / darwin-build (push) Has been cancelled
release / windows-build (amd64, x86_64, windows) (push) Has been cancelled
release / windows-build (arm64, aarch64, windows) (push) Has been cancelled
release / windows-app (push) Has been cancelled
release / linux-build (amd64, linux, archive) (push) Has been cancelled
release / linux-build (amd64, linux, rocm) (push) Has been cancelled
release / linux-build (arm64, linux, archive) (push) Has been cancelled
release / docker-build-push (amd64, CGO_CFLAGS
CGO_CXXFLAGS
GOFLAGS
, linux) (push) Has been cancelled
release / docker-build-push (amd64, CGO_CFLAGS
CGO_CXXFLAGS
GOFLAGS
FLAVOR=rocm
, linux, -rocm) (push) Has been cancelled
release / docker-build-push (arm64, CGO_CFLAGS
CGO_CXXFLAGS
GOFLAGS
, linux) (push) Has been cancelled
release / docker-merge-push () (push) Has been cancelled
release / docker-merge-push (-rocm) (push) Has been cancelled
release / release (push) Has been cancelled
* docs: vulkan information
* Revert "CI: Set up temporary opt-out Vulkan support (#12614)"
This reverts commit 8b6e5baee7.
* vulkan: temporary opt-in for Vulkan support
Revert this once we're ready to enable by default.
* win: add vulkan CI build
92 lines
2.4 KiB
Plaintext
92 lines
2.4 KiB
Plaintext
## CPU only
|
|
|
|
```shell
|
|
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
|
|
```
|
|
|
|
## Nvidia GPU
|
|
|
|
Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation).
|
|
|
|
### Install with Apt
|
|
|
|
1. Configure the repository
|
|
|
|
```shell
|
|
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
|
|
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
|
|
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
|
|
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
|
|
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
|
|
sudo apt-get update
|
|
```
|
|
|
|
2. Install the NVIDIA Container Toolkit packages
|
|
|
|
```shell
|
|
sudo apt-get install -y nvidia-container-toolkit
|
|
```
|
|
|
|
### Install with Yum or Dnf
|
|
|
|
1. Configure the repository
|
|
|
|
```shell
|
|
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
|
|
| sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
|
|
```
|
|
|
|
2. Install the NVIDIA Container Toolkit packages
|
|
|
|
```shell
|
|
sudo yum install -y nvidia-container-toolkit
|
|
```
|
|
|
|
### Configure Docker to use Nvidia driver
|
|
|
|
```shell
|
|
sudo nvidia-ctk runtime configure --runtime=docker
|
|
sudo systemctl restart docker
|
|
```
|
|
|
|
### Start the container
|
|
|
|
```shell
|
|
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
|
|
```
|
|
|
|
<Note>
|
|
If you're running on an NVIDIA JetPack system, Ollama can't automatically discover the correct JetPack version.
|
|
Pass the environment variable `JETSON_JETPACK=5` or `JETSON_JETPACK=6` to the container to select version 5 or 6.
|
|
</Note>
|
|
|
|
## AMD GPU
|
|
|
|
To run Ollama using Docker with AMD GPUs, use the `rocm` tag and the following command:
|
|
|
|
```shell
|
|
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm
|
|
```
|
|
|
|
## Vulkan Support
|
|
|
|
Vulkan is bundled into the `ollama/ollama` image.
|
|
|
|
```shell
|
|
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 -e OLLAMA_VULKAN=1 --name ollama ollama/ollama
|
|
```
|
|
|
|
|
|
## Run model locally
|
|
|
|
Now you can run a model:
|
|
|
|
```shell
|
|
docker exec -it ollama ollama run llama3.2
|
|
```
|
|
|
|
## Try different models
|
|
|
|
More models can be found on the [Ollama library](https://ollama.com/library).
|
|
|