Re: llama.cpp vulkan

Hi,

Seems to work with my Radeon card.

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 6900 XT (RADV NAVI21) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none

llama-cli -m ./mistral-7b-instruct-v0.2.Q8_0.gguf
[..]
llama_perf_sampler_print: sampling time = 7.18 ms / 235 runs ( 0.03 ms per token, 32720.69 tokens per second)
llama_perf_context_print: load time = 1298.46 ms
llama_perf_context_print: prompt eval time = 4730.92 ms / 23 tokens ( 205.69 ms per token, 4.86 tokens per second)
llama_perf_context_print: eval time = 33557.39 ms / 221 runs ( 151.84 ms per token, 6.59 tokens per second)
llama_perf_context_print: total time = 157345.06 ms / 244 tokens


Using ngl to offload some layers to GPU :

llama-cli -m ./mistral-7b-instruct-v0.2.Q8_0.gguf -ngl 32
[...]
llama_perf_sampler_print: sampling time = 7.44 ms / 257 runs ( 0.03 ms per token, 34543.01 tokens per second)
llama_perf_context_print: load time = 2210.50 ms
llama_perf_context_print: prompt eval time = 3417.91 ms / 22 tokens ( 155.36 ms per token, 6.44 tokens per second)
llama_perf_context_print: eval time = 4990.24 ms / 244 runs ( 20.45 ms per token, 48.90 tokens per second)
llama_perf_context_print: total time = 11833.75 ms / 266 tokens



On Sat, Feb 1, 2025, at 05:31, Chris Cappuccio wrote:
> This might be a way to do it. Does anyone have a card to test against?
>
> Index: Makefile
> ===================================================================
> RCS file: /cvs/ports/misc/llama.cpp/Makefile,v
> retrieving revision 1.1
> diff -u -p -u -r1.1 Makefile
> --- Makefile 30 Jan 2025 22:55:11 -0000 1.1
> +++ Makefile 1 Feb 2025 04:20:02 -0000
> @@ -10,6 +10,7 @@ SHARED_LIBS += ggml-cpu 0.0
> SHARED_LIBS += ggml 0.0
> SHARED_LIBS += llama 0.0
> SHARED_LIBS += llava_shared 0.0
> +SHARED_LIBS += ggml-vulkan 0.0
>
> CATEGORIES = misc
>
> @@ -18,11 +19,15 @@ HOMEPAGE = https://github.com/ggerganov
> # MIT
> PERMIT_PACKAGE = Yes
>
> -WANTLIB += m pthread ${COMPILER_LIBCXX}
> +WANTLIB += m pthread vulkan ${COMPILER_LIBCXX}
>
> MODULES = devel/cmake
>
> +LIB_DEPENDS = graphics/vulkan-loader
> +BUILD_DEPENDS = graphics/shaderc
> +
> CONFIGURE_ARGS = -DGGML_CCACHE=Off \
> - -DGGML_NATIVE=Off
> + -DGGML_NATIVE=Off \
> + -DGGML_VULKAN=On
>
> .include <bsd.port.mk>
> Index: pkg/PLIST
> ===================================================================
> RCS file: /cvs/ports/misc/llama.cpp/pkg/PLIST,v
> retrieving revision 1.1
> diff -u -p -u -r1.1 PLIST
> --- pkg/PLIST 30 Jan 2025 22:55:11 -0000 1.1
> +++ pkg/PLIST 1 Feb 2025 04:20:02 -0000
> @@ -58,6 +58,7 @@ bin/convert_hf_to_gguf.py
> @bin bin/test-tokenizer-0
> @bin bin/test-tokenizer-1-bpe
> @bin bin/test-tokenizer-1-spm
> +@bin bin/vulkan-shaders-gen
> include/ggml-alloc.h
> include/ggml-backend.h
> include/ggml-blas.h
> @@ -74,7 +75,6 @@ include/ggml.h
> include/gguf.h
> include/llama-cpp.h
> include/llama.h
> -lib/cmake/
> lib/cmake/ggml/
> lib/cmake/ggml/ggml-config.cmake
> lib/cmake/ggml/ggml-version.cmake
> @@ -83,6 +83,7 @@ lib/cmake/llama/llama-config.cmake
> lib/cmake/llama/llama-version.cmake
> @lib lib/libggml-base.so.${LIBggml-base_VERSION}
> @lib lib/libggml-cpu.so.${LIBggml-cpu_VERSION}
> +@lib lib/libggml-vulkan.so.${LIBggml-vulkan_VERSION}
> @lib lib/libggml.so.${LIBggml_VERSION}
> @lib lib/libllama.so.${LIBllama_VERSION}
> @lib lib/libllava_shared.so.${LIBllava_shared_VERSION}