The Problem with Local AI on AMD Hardware
Running models locally with tools like Ollama or LM Studio on AMD GPUs often leads to two roadblocks:
Missing drivers or unrecognised GPU architectures requiring HSA_OVERRIDE_GFX_VERSION manual tweaks.
Uncertainty over whether to run Vulkan backends or ROCm/HIP backends for maximum tokens per second.
Solution 1: ROCmFix
ROCmFix is a single-file Python utility designed to resolve HSA_OVERRIDE_GFX_VERSION issues automatically.
Key Features:
Query PCI IDs directly using Windows Registry or Linux lspci.
Set permanent or session environment variables for CMD, PowerShell, Bash, Zsh, and Fish.
Run rocmfix doctor to inspect installed HIP SDK and Vulkan components.
# Quick run
python rocmfix.py
Solution 2: InferBench
InferBench automates testing local LLM inference speeds across backend engines.
Benchmarking Workflow:
Executes warm-up queries.
Forces VRAM unloads between runs to eliminate memory fragmentation/caching bias.
Calculates median tok/s and TTFT (Time-to-First-Token).
Check out the repositories on GitHub:
