How to Run Unsloth on AMD GPUs (ROCm) [2026 Guide]

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

You don’t need another “it installed!” ROCm post. You need Unsloth fine-tuning on an AMD GPU to actually train, with a loss curve that goes down for the right reasons.

By the end of this guide, you’ll have Unsloth fine-tuning running on an AMD GPU with ROCm, plus a quick sanity run that proves it’s converging (not just “it launched”). If your setup is close to supported, you can get from zero to a minimal LoRA run in ~45–90 minutes. If you’ve been stuck in “HIP error / no kernel image available” hell, plan for an afternoon.

This is an Unsloth AMD GPU ROCm fine tuning guide written for late‑2026 reality: ROCm support is way better than it was, but it’s still version-fragile, kernel-fragile, and “works” is not the same as “trains fast and converges”.

I’m going to be blunt. Most AMD fine-tuning failures aren’t because Unsloth is “broken”. They’re because your stack is a Jenga tower: kernel + driver + ROCm userland + PyTorch ROCm wheels + Triton/attention kernels + mixed precision. One mismatch and you get either a hard crash or, worse, silent slow fallbacks.

(Visual break: matrix screenshot)

What is Unsloth?

Unsloth is a training and inference optimization toolkit that patches Hugging Face Transformers/TRL flows to make fine-tuning large language models faster and more memory-efficient, typically using LoRA and related parameter-efficient methods.

Where Unsloth earns its keep is (a) smart model patching and (b) optional high-performance kernels. On NVIDIA/CUDA, those kernels are usually just… there. On AMD/ROCm, kernel availability depends on your exact GPU architecture and software versions. That’s the whole game.

If you’re still deciding whether you need fine-tuning at all, read my decision framework on fine-tuning vs RAG (retrieval-augmented generation) and prompt work.

Unsloth AMD GPU ROCm fine tuning guide: the late‑2026 compatibility matrix

Here’s the matrix I wish existed the first time I tried to help someone debug Unsloth on a Radeon card. It’s not “official”. It’s the set of combinations that tend to behave in the real world, plus the landmines that waste your time.

Two anchors before we start:

• Treat the PyTorch ROCm wheel selector as the source of truth. PyTorch’s install page currently shows Stable PyTorch 2.7.0 and ROCm wheel availability for ROCm 6.3 (that compute platform option exists right in the selector). If your installed ROCm runtime is 6.1 and you pip-installed a 6.3 wheel, you’re basically asking for missing hip libraries and runtime weirdness. See the official selector on PyTorch.
• ROCm itself ships a published compatibility matrix and install guidance. Start from AMD’s docs, not random Reddit incantations. AMD ROCm documentation has an explicit “Compatibility matrix” section and per-component pages.

Known-good stacks (pin these first)

“Known-good” means one thing: you can run a small SFT, loss decreases smoothly, no NaNs, and throughput isn’t obviously crippled. Anything else is a science project.

A few sharp edges I’ll save you from:

• If you’re on Windows: ROCm exists there, but Unsloth fine-tuning is still dramatically less predictable than on Linux. If your goal is “fine-tune this weekend”, use Linux.
• If you’re on WSL2: it’s great for a lot of dev. For ROCm fine-tuning, you’re stacking two “almost works” layers. If you want my honest recommendation: don’t.

Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, the pattern I keep seeing across non-NVIDIA platforms is: getting the model to load is easy; getting stable throughput is the real work. ROCm fine-tuning is the same story.

(Visual break: “known good” command snippet screenshot)

Install ROCm and verify HIP works (rocminfo/hipcc)

This section is intentionally boring. Boring is the point. If you skip verification, you’ll spend the next 6 hours blaming Unsloth for a driver/runtime problem.

1) Pick a supported OS and kernel first

Start with a mainstream distro ROCm expects. In 2026, Ubuntu LTS is the pragmatic choice.

• Ubuntu 24.04 LTS is a solid baseline.
• Linux kernel 6.x is common.

The moment you go off-road (custom kernels, odd GCC/libstdc++ combos), you’re volunteering to debug C++ ABI errors instead of training models.

2) Install ROCm from AMD’s docs

Use AMD’s official install steps and their compatibility matrix: AMD ROCm documentation.

Practical advice I stick to:

• Keep your ROCm install single-source. Don’t mix distro packages + random tarballs + Docker bind mounts unless you know exactly why.
• If you’re using containers (often best for Radeon), make sure the host driver stack matches what the container expects. ROCm is not magic.

3) Verify rocminfo and hipcc

After install:

# should print GPU + agent info
/opt/rocm/bin/rocminfo | head -n 50

# should print ROCm/HIP compiler info
/opt/rocm/bin/hipcc --version

If rocminfo can’t see your GPU, stop. Don’t “try pip installs until it works”. Fix the driver/runtime first.

Common failure modes:

• rocminfo: HSA runtime not found → ROCm userland isn’t installed correctly or not on PATH.
• Permission issues accessing /dev/kfd → user not in the right group (often video/render).

Install PyTorch for ROCm (matching wheel to ROCm)

This is the #1 root cause of “it imports, but training explodes”. The PyTorch ROCm wheel bakes assumptions about the ROCm runtime.

The only rule that matters

Your installed ROCm runtime version and your PyTorch ROCm build must match.

PyTorch makes this easy if you actually use their selector. On the official page, the compute platform dropdown includes ROCm 6.3 and generates the matching --index-url command. That selector is maintained by the people shipping the wheels. Use it: PyTorch.

Example (ROCm wheel install style):

python3 -m venv .venv
source .venv/bin/activate

# example only. generate your exact command from pytorch.org
pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3

Verify PyTorch sees the AMD GPU

python - <<'PY'
import torch
print('torch', torch.__version__)
print('hip available', torch.cuda.is_available())
print('device', torch.cuda.get_device_name(0) if torch.cuda.is_available() else None)
PY

Yes, it still prints cuda in APIs. No, you’re not secretly using NVIDIA.

If is_available() is false:

• You installed a CPU wheel.
• Or you installed a ROCm wheel that doesn’t match your ROCm runtime.

If you see linker errors about missing libamdhip64.so or friends, you’re mismatched.

Set up Transformers + TRL + Unsloth

Unsloth rides on top of Hugging Face Transformers and TRL.

• Unsloth repo: Unsloth AI
• TRL docs for trainer knobs (packing, clipping, etc.): TRL

My opinionated dependency strategy for AMD

On CUDA I’m okay with “latest-ish” because the ecosystem is brutally well-tested.

On ROCm I pin. Always.

Pin your layers like you mean it:

• Pin torch to the ROCm wheel you installed.
• Pin transformers, trl, accelerate, peft to a tested set.
• Upgrade only one layer at a time.

Example:

pip install "transformers==4.56.*" "trl==0.12.*" "accelerate==1.0.*" "peft==0.13.*"

# install unsloth
pip install "unsloth[rocm]" || pip install unsloth

If the extra [rocm] doesn’t exist in the current Unsloth packaging when you read this, that’s fine. The point is: install Unsloth, then verify what optional kernels it pulled in.

Attention kernels: assume SDPA first

One of the most common traps is people assuming FlashAttention is mandatory because every NVIDIA guide reads like it’s oxygen.

On AMD, start with SDPA (PyTorch scaled dot-product attention). Chase FlashAttention only after you have a converging baseline.

Why I’m so stubborn about this:

• FlashAttention availability on ROCm is architecture- and version-dependent.
• The failure mode is often not “missing package”. It’s “Triton compiled something, ran it, and now you get NaNs at step 200.” That’s the worst kind of failure.

Run a minimal LoRA/SFT sanity check (prove it converges)

If you only do one thing from this post, do this. Don’t launch your full dataset run first. That’s how you burn an evening.

The 5-step checklist (minimal moving parts)

1. Confirm rocminfo sees the GPU.
2. Confirm PyTorch ROCm sees the GPU (torch.cuda.is_available() is true).
3. Run a single-batch forward+backward in fp16.
4. Run 50–200 training steps on a tiny dataset.
5. Confirm loss decreases smoothly and no NaNs appear.

That’s your “stack is sane” certificate.

Minimal TRL SFTTrainer script (ROCm-friendly defaults)

This uses conservative knobs: fp16, gradient checkpointing, modest learning rate, clipping.

# sft_rocm_sanity.py
import os
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")

import torch
from datasets import Dataset
from transformers import AutoTokenizer
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig

from unsloth import FastLanguageModel

model_name = "meta-llama/Meta-Llama-3-8B-Instruct"  # 8B class is a sane starting point

# Tiny dataset on purpose
data = Dataset.from_list([
  {"text": "Write a haiku about GPUs."},
  {"text": "Explain what ROCm is in one paragraph."},
  {"text": "Summarize the difference between LoRA and full fine-tuning."},
] * 200)

max_seq_length = 512

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=model_name,
    max_seq_length=max_seq_length,
    dtype=torch.float16,
    load_in_4bit=False,   # start without QLoRA on AMD
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=data,
    peft_config=lora_config,
    args=SFTConfig(
        output_dir="./out_rocm_sanity",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=2e-4,
        warmup_ratio=0.03,
        max_steps=200,
        logging_steps=10,
        fp16=True,
        bf16=False,
        gradient_checkpointing=True,
        max_grad_norm=1.0,
        packing=False,
        report_to=[],
    ),
)

trainer.train()
print("done")

Run it:

python sft_rocm_sanity.py

What “good” looks like:

• Loss starts high and trends down over the first 50–200 steps.
• No nan loss. No sudden spikes to absurd values.
• GPU utilization is non-zero and stable.

If you get NaNs, don’t immediately blame ROCm. Do the boring trainer triage first:

• Lower LR from 2e-4 → 1e-4.
• Ensure max_grad_norm=1.0 is set.
• Turn off packing.

TRL documents these knobs and what they change: TRL.

Troubleshoot common ROCm errors (HIP runtime, missing libs, kernel image issues)

Most guides turn this section into an error-message graveyard. That’s useless. You want a decision tree.

1) “HIP error / invalid device function / no kernel image available”

Translation: a compiled kernel doesn’t match your GPU architecture, or you installed a build that simply wasn’t compiled for your card.

Fix path:

• Confirm your GPU is actually supported by your ROCm version in AMD’s matrix: AMD ROCm documentation.
• Confirm your PyTorch wheel matches your ROCm runtime. Reinstall PyTorch using the official selector: PyTorch.
• If this happens when enabling FlashAttention/Triton kernels: force SDPA and disable fused attention paths.

Useful env knobs to try (these move around across versions, so treat this as a debugging tactic, not a forever setting):

export TORCH_LOGS="+dynamo,+inductor"
export TORCHDYNAMO_VERBOSE=1

If you see Triton compilation errors referencing HIP, that’s your sign to stop chasing fancy kernels and get a clean SDPA baseline.

2) Missing hipblaslt / rccl / shared library errors

This is almost always packaging mismatch:

• ROCm userland not fully installed.
• Container image expects a different ROCm stack than the host.

My stance: containers are usually more reliable on Radeon, because you can pin userland and Python deps together. But your host driver still matters.

If you want the broader ROCm vs CUDA story for local AI, I wrote that comparison already. This post is narrower: making Unsloth fine-tuning stop failing.

3) bitsandbytes fails on ROCm (4-bit/8-bit)

This question shows up constantly: “Why does bitsandbytes 4-bit/8-bit quantization fail on ROCm?”

Because bitsandbytes has historically been CUDA-first. ROCm support has improved, but it’s still fragile, and it lags the newest training stacks.

Practical alternatives that don’t waste your weekend:

• FP16 LoRA (the baseline I recommend in this post).
• 8-bit optimizers only if your ROCm build supports the exact kernel path you need.
• If your goal is cheap inference, skip training quantization entirely. Use GGUF for inference-only. I’ve written about quantization tradeoffs in LLM quantization levels and GGUF vs GPTQ vs EXL2.

If you’re on a 16GB Radeon card and trying to QLoRA a 13B model because TikTok told you it’s “easy”, stop. Use a smaller base model or add VRAM.

4) “It runs” but throughput is awful (kernel fallbacks)

This is the sneaky failure. No crash. No obvious error. Just performance that feels like you’re training through a straw.

Symptoms:

• GPU utilization is low.
• Tokens/sec is way lower than expected.
• You see unexpected CPU usage spikes.

How to diagnose:

• Turn on PyTorch compile logs (TORCH_LOGS above) to see if Inductor/Triton is bailing.
• Start from SDPA. If FlashAttention “works” but is slower, you’re probably hitting a fallback path.

Also: measure something real.

In my benchmark work for this site, throughput is the metric people hand-wave the most. That’s why I keep pushing tok/s and TTFT in posts like local LLM benchmark methodology.

5) NaNs, divergence, and fake convergence

On AMD you’ll sometimes get “training” that doesn’t crash but is numerically unstable.

Things I check, in this order:

• Loss: do you ever see nan? If yes, don’t continue.
• Gradient clipping: set max_grad_norm to 1.0 (or even 0.5).
• Precision: prefer fp16 first. bf16 can be great, but bf16 stability varies by GPU + stack.
• Sequence packing: turn it off until you’re stable.

Rule of thumb VRAM budgets (practical, not theoretical):

• 7B/8B LoRA: 16GB can work with small batch, seq len ~512–1024, grad checkpointing.
• 13B LoRA: 24GB is the sane starting point for non-suffering.
• Anything beyond: either multi-GPU (more pain on Radeon) or change strategy.

If you want the bigger hardware sizing story for local LLM work, I have a dedicated guide. This post stays focused on Unsloth + ROCm.

Here’s an official baseline walkthrough from AMD itself. It’s worth watching once to calibrate your environment expectations:

[YOUTUBE:GNF32XuRAN4|How to Fine-Tune a Model on AMD GPUs using LoRA]

Container vs bare-metal for Radeon: my default recommendation

If you’re on consumer Radeon, I generally recommend containers for fine-tuning.

Reasons:

• You can pin Python deps and ROCm userland together.
• You can share a known-good image with your team.
• You reduce “my system Python upgraded libstdc++ and now hip runtime crashes” incidents.

Bare-metal is fine too if you’re disciplined. Most people aren’t.

If you do bare-metal:

• Keep a dedicated venv per ROCm/PyTorch stack.
• Don’t upgrade blindly.

This mindset is similar to what I preach for reproducible dev envs in general, like in my Python dev setup and Linux migration plan posts.

My prediction for 2027: “ROCm works” won’t be enough

By 2027, “can I run it?” will be boring. It’ll be solved for AMD consumer GPUs.

The dividing line will be: can you update your stack without accidentally losing fast kernels, falling back to slow paths, or reintroducing numerical weirdness? NVIDIA has earned that kind of stability. AMD is getting closer, but Unsloth-style optimization stacks are exactly where the cracks still show.

If you’re building a serious fine-tuning workflow on Radeon today, don’t treat this as a one-off setup. Turn it into a pinned, scripted environment you can recreate on a fresh machine. That’s how you stop ROCm from becoming a recurring calendar event.

Originally published on kunalganglani.com