Re: llama.cpp vulkan

Seems to work with

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 6600 (RADV NAVI23) (radv) | uma: 0 |
fp16: 1 | warp size: 64 | matrix cores: none

Not a proper benchmark, but with -ngl 32, responding happens *a lot*
faster, and I've heard my GPU fan for I think the first time ever.

The only downside is that running llama-cli with a user that doesn't
have access to the GPU now fails with

libc++abi: terminating due to uncaught exception of type
vk::InitializationFailedError: vk::Instance::enumeratePhysicalDevices:
ErrorInitializationFailed

ok volker@

On 2/1/25 5:31 AM, Chris Cappuccio wrote:
> This might be a way to do it. Does anyone have a card to test against?
>
> Index: Makefile
> ===================================================================
> RCS file: /cvs/ports/misc/llama.cpp/Makefile,v
> retrieving revision 1.1
> diff -u -p -u -r1.1 Makefile
> --- Makefile 30 Jan 2025 22:55:11 -0000 1.1
> +++ Makefile 1 Feb 2025 04:20:02 -0000
> @@ -10,6 +10,7 @@ SHARED_LIBS += ggml-cpu 0.0
> SHARED_LIBS += ggml 0.0
> SHARED_LIBS += llama 0.0
> SHARED_LIBS += llava_shared 0.0
> +SHARED_LIBS += ggml-vulkan 0.0
>
> CATEGORIES = misc
>
> @@ -18,11 +19,15 @@ HOMEPAGE = https://github.com/ggerganov
> # MIT
> PERMIT_PACKAGE = Yes
>
> -WANTLIB += m pthread ${COMPILER_LIBCXX}
> +WANTLIB += m pthread vulkan ${COMPILER_LIBCXX}
>
> MODULES = devel/cmake
>
> +LIB_DEPENDS = graphics/vulkan-loader
> +BUILD_DEPENDS = graphics/shaderc
> +
> CONFIGURE_ARGS = -DGGML_CCACHE=Off \
> - -DGGML_NATIVE=Off
> + -DGGML_NATIVE=Off \
> + -DGGML_VULKAN=On
>
> .include <bsd.port.mk>
> Index: pkg/PLIST
> ===================================================================
> RCS file: /cvs/ports/misc/llama.cpp/pkg/PLIST,v
> retrieving revision 1.1
> diff -u -p -u -r1.1 PLIST
> --- pkg/PLIST 30 Jan 2025 22:55:11 -0000 1.1
> +++ pkg/PLIST 1 Feb 2025 04:20:02 -0000
> @@ -58,6 +58,7 @@ bin/convert_hf_to_gguf.py
> @bin bin/test-tokenizer-0
> @bin bin/test-tokenizer-1-bpe
> @bin bin/test-tokenizer-1-spm
> +@bin bin/vulkan-shaders-gen
> include/ggml-alloc.h
> include/ggml-backend.h
> include/ggml-blas.h
> @@ -74,7 +75,6 @@ include/ggml.h
> include/gguf.h
> include/llama-cpp.h
> include/llama.h
> -lib/cmake/
> lib/cmake/ggml/
> lib/cmake/ggml/ggml-config.cmake
> lib/cmake/ggml/ggml-version.cmake
> @@ -83,6 +83,7 @@ lib/cmake/llama/llama-config.cmake
> lib/cmake/llama/llama-version.cmake
> @lib lib/libggml-base.so.${LIBggml-base_VERSION}
> @lib lib/libggml-cpu.so.${LIBggml-cpu_VERSION}
> +@lib lib/libggml-vulkan.so.${LIBggml-vulkan_VERSION}
> @lib lib/libggml.so.${LIBggml_VERSION}
> @lib lib/libllama.so.${LIBllama_VERSION}
> @lib lib/libllava_shared.so.${LIBllava_shared_VERSION}
>