How to Write a Kernel Bug Report Maintainers Act On

Originally published on the TECH VEDA blog: How to Write a Kernel Bug Report Maintainers Act On

A kernel bug report gets acted on when it reaches the right maintainer, on a kernel that maintainer can still change, with the failure decoded and the reproduction written down. Most reports that are ignored fail on one of those four points, not on the quality of the bug. The work of getting them right is ordinary engineering work, and it is visible in a public archive long after you send it.

Reporting a fault you found is one of the few pieces of engineering work that carries your name outside your company. A kernel bug report sent to a mailing list is archived on lore.kernel.org permanently, read by the people who maintain the code you depend on, and often cited in the commit that fixes it. A precise report gets a reply and sometimes a Reported-by: line; a vague one is skipped, because the maintainer has no way to act on it. The difference is almost never the difficulty of the bug. It is preparation.

Why a kernel bug report is a career artifact

There is no central bug tracker for the Linux kernel. As the kernel's own reporting guide puts it, issues "typically need to be sent by mail to a maintainer and a public mailing list". Your report is therefore not a ticket in a queue that someone is paid to process. It is a message to a named engineer, competing with everything else in that engineer's inbox, and it remains permanently archived. When a developer fixes the problem, Documentation/process/submitting-patches.rst tells them to credit you with a Reported-by: tag and a Closes: tag pointing at the URL of your report, and private trackers are not allowed there. A report on an archived list can be cited by the fix; one in an internal tracker cannot. Over a few years that becomes a public record of problems you found and described well, which is a harder thing to build than a list of technologies.

Before you send a kernel bug report

The rest of this article works through each of these steps in order. If you have sent a kernel bug report before, the list on its own is the working checklist.

Reproduce the fault on a kernel upstream can still fix, which normally means current mainline.
Confirm the kernel is not tainted, and reboot if it is.
Decode the stack trace against the matching vmlinux.
Find the maintainers and lists with scripts/get_maintainer.pl.
Reduce the reproduction to the shortest sequence that still triggers the fault.
Collect the version, hardware and log details the report must carry.
If the fault is a regression, add the regression route: subject prefix, regressions list, regzbot line, ideally a bisected commit.
If the fault has security impact, stop and use the private route instead.

Send it to the right people, not to a tracker

A kernel bug report has no default destination, so this is the step that decides whether anyone reads it. Find the subsystem, then find its maintainers. The MAINTAINERS file lists them with M: for mail, L: for mailing list, S: for status, and B: where a subsystem uses a bug tracker. Very few subsystems use one, and only some of those use bugzilla.kernel.org. The script in the tree does the lookup for you:

raghu@techveda.org:~$ ./scripts/get_maintainer.pl -f drivers/net/wireless/ath/ath10k*
Some Human <shuman@example.com> (supporter:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER)
Another S. Human <asomehuman@example.com> (maintainer:NETWORKING DRIVERS)
ath10k@lists.infradead.org (open list:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER)
linux-wireless@vger.kernel.org (open list:NETWORKING DRIVERS (WIRELESS))
netdev@vger.kernel.org (open list:NETWORKING DRIVERS)
linux-kernel@vger.kernel.org (open list)

Do not mail everyone the script prints. Address the maintainers, then Cc the most specific mailing list for that code and the Linux Kernel Mailing List. If you are unsure which file the fault sits in, identify the driver first:

raghu@techveda.org:~$ lspci -k
3a:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
    Subsystem: Bigfoot Networks, Inc. Device 1535
    Kernel driver in use: ath10k_pci
    Kernel modules: ath10k_pci

One exception overrides all of this. If the fault has security consequences, do not post it to a public list at all. Send it privately to the kernel security team at security@kernel.org, in plain text with no attachments, and state the affected kernel version range, because reports without a version range are not processed.

Reproduce on a kernel upstream can still fix

Once the report is routed correctly, the next question is whether upstream can act on the kernel you tested. Upstream developers cannot fix a kernel they did not write. If you are running a vendor or distribution kernel with vendor patches on top, the reporting guide is direct: report it to the vendor, not upstream. A warranty or support contract with a vendor does not entitle you to fixes from upstream developers.

For an upstream report, install a mainline kernel. Most of the time kernel.org's front page shows mainline as a pre-release such as 7.2-rc7, and that is the one to use, because fixes must be applied there first. Do not avoid it because of the "rc". The latest stable release is acceptable in some situations and is the better choice during a merge window. Longterm kernels are unsuitable for this first report, being too far from current code; a fault that exists only inside a longterm series goes to the stable list instead. Whichever you pick, use a vanilla build: sources taken straight from kernel.org, not modified or enhanced.

Check the taint flag before you write anything

A clean upstream kernel still does not make a kernel bug report trustworthy if the system is tainted. The operational rule is simple: reboot into a clean, untainted kernel before you collect any evidence you intend to send. The kernel sets a taint flag when something has happened that can cause unrelated follow-up errors, and a report from a tainted kernel is frequently set aside, because the maintainer cannot tell whether your fault is the real bug or a consequence of something else.

raghu@techveda.org:~$ cat /proc/sys/kernel/tainted
0

Zero means the kernel is clean. Any other number is a bitfield of reasons, and the script tools/debugging/kernel-chktaint decodes it for you; distributions usually ship it in a linux-tools or kernel-tools package.

In a crash log the state appears on the line starting with CPU:, after the process id and command name:

CPU: 0 PID: 4424 Comm: insmod Tainted: P        W  O      4.20.0-0.rc6.fc30 #1

You will see Not tainted there if the kernel was clean, or Tainted: followed by letters and blanks if it was not. These are the letters that matter most in a report:

Letter	Meaning
`P`	A proprietary module was loaded. Any module whose licence `insmod` does not recognise as GPL compatible counts.
`O`	An externally built, out-of-tree module was loaded.
`E`	An unsigned module was loaded on a kernel that supports module signatures.
`C`	A staging driver was loaded. Acceptable if that staging driver is the subject of your report.
`D`	The kernel died recently, meaning an earlier Oops or BUG. In the log, `[#1]` marks the first Oops since boot; anything after it may be a consequence of it.
`W`	The kernel issued a warning earlier.

In the line above, the kernel was tainted because a proprietary module was loaded, a warning occurred, and an out-of-tree module was loaded. Two practical notes. The taint stays set even after you unload whatever caused it, so clearing it needs a reboot and not just an rmmod. And DKMS and akmods rebuild external modules automatically, so disable them for the test, then check the flag again on the freshly installed kernel, because that is the kernel your report is about.

Decode the failure instead of pasting it

A stack trace full of hexadecimal offsets is work you are asking the maintainer to do. Do it yourself. Build with CONFIG_KALLSYMS, CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO enabled, then pass the log through the decoder in the tree with the matching vmlinux:

raghu@techveda.org:~$ sudo dmesg | ./linux-src/scripts/decode_stacktrace.sh ./linux-src/vmlinux

That turns a line like this:

[   68.387301] RIP: 0010:test_module_init+0x5/0xffa [test_module]

into a line naming the source file and line number, which is what a maintainer actually reads. This is the same decoding you would do for a driver that fails during probe, so the skill transfers directly from internal debugging to upstream reporting.

Regressions take a different route

Once the trace is decoded, decide whether the fault is a regression, because regressions follow a different route. A regression is a case where something that worked on one kernel works worse or not at all on a newer one built with a similar configuration. That rule is the first rule of kernel development, and it covers only the interfaces the kernel provides to user space, so breakage of kernel-internal interfaces used by out-of-tree modules does not qualify. A slowdown can qualify, but only a significant one: five percent in a micro-benchmark is not enough unless a broad benchmark also moves by more than one percent.

Regressions are handled with more urgency and need extra steps. Start the subject with [REGRESSION]. Cc the regressions mailing list at regressions@lists.linux.dev. If it broke inside a stable series, for example between 7.1.4 and 7.1.5, Cc stable@vger.kernel.org as well. Then tell the tracking bot about it by putting a line like this in its own paragraph, with a blank line above and below:

#regzbot introduced: v7.1..v7.2-rc1

The first version still worked and the second is where the problem appeared. Better still, run git bisect and give the commit id instead of the range. That means building ten to twenty kernels, but developers of the affected code often cannot bisect it themselves because they do not have your hardware, so it is the work that most improves a regression report. Add the author of that commit and its Signed-off-by chain to the recipients. Registering the regression matters because Linus Torvalds uses the bot's weekly reports when deciding whether a release is ready, so an untracked regression may not be counted at all.

Write the report from evidence to subject

Only after the evidence is complete should the summary and the subject be written. Developers skim, and the subject, the first sentence and the first paragraph decide whether the rest is read, so those are the last things you write. Write the detailed description first, then a normal paragraph on top summarising the problem and its impact, then one sentence above that, then a short descriptive subject.

A finished kernel bug report has a predictable shape. This skeleton is illustrative rather than a real report, but the order and the headings are what a maintainer expects to find:

Subject: [REGRESSION] ath10k: connection drops after resume from suspend

On v7.2-rc7 the QCA6174 loses its association after resume from suspend
and does not recover without unloading and reloading the module. The same
hardware and the same configuration work on v7.1. Bisection points at
<commit id>, whose author is on Cc.

Environment
  Kernel:   (output of cat /proc/version)
  Arch:     (output of uname -mi)
  Distro:   (Operating System line, or /etc/os-release)
  Hardware: (exact model and revision)
  Taint:    Not tainted

Steps to reproduce
  1. Boot and associate with a WPA2 access point.
  2. Suspend to RAM.
  3. Resume.

Expected result
  The association survives resume.

Actual result
  The interface reports no carrier. Decoded trace and full dmesg linked
  below.

Logs
  dmesg:   (link)
  .config: (link)

#regzbot introduced: v7.1..v7.2-rc7

The regression-specific parts, meaning the subject prefix and the regzbot line, drop out for an ordinary bug. Everything else stays.

Every report should carry these:

cat /proc/version, so the exact kernel is unambiguous.
hostnamectl | grep "Operating System" and uname -mi for distribution and architecture. On a system without systemd, which is common on embedded targets, read /etc/os-release instead.
The exact hardware model. "Dell XPS 13" is not enough; the model number such as 9380 is.
A reproduction procedure, reduced to the shortest sequence that still triggers the fault.
The full dmesg, starting at the "Linux version" line. If that line is missing, early messages were discarded, so use journalctl -b 0 -k instead.

Do not attach dmesg and .config to the mail; they make it too large for the list. Upload them somewhere stable that will still resolve in a year, and put the links in the mail. Report one issue per mail, because different issues go to different developers. And do not report a fault that happened once and never again.

One caution if the board belongs to your employer or a customer: a kernel log can carry hostnames, serial numbers and unreleased hardware identifiers. Read it before you publish it, and agree internally on what may go to a public archive. Removing a hostname is acceptable; editing the fault itself is not. If the sensitive part cannot be separated from the evidence, reproduce the problem on hardware you can discuss openly.

After you send the report

Sending it starts a conversation rather than closing a submission. Expect one to five business days for a reply, and longer during a merge window or a conference. Maintainers should respond to every report, but they are only obliged to fix the high priority ones: regressions, security issues, and really severe problems such as a panic. For an ordinary report, wait two to three weeks before a friendly reminder, three more weeks before a second one, and only then ask a higher-level maintainer for advice.

Your side of the exchange matters as much. Reply in public with reply-all, answer test requests within a few business days, and retest with each new release candidate and report what you found. That is what turns one report into a working relationship with a subsystem, which is the part that compounds over a career.

Key takeaways

There is no central tracker. A kernel bug report is mail to a named maintainer, with the specific list and LKML in Cc.
Report vendor kernel faults to the vendor. Report upstream faults against a vanilla mainline kernel, usually the current release candidate.
Check the taint flag on the kernel you are reporting about, and remove external modules before testing.
Decode the stack trace yourself; an undecoded trace transfers your work to the maintainer.
Regressions need a [REGRESSION] subject, the regressions list in Cc, and a #regzbot introduced: line. A bisected commit id is worth the build time.
Security faults never go to a public list. They go privately to security@kernel.org with the affected version range.
Write the body first and the subject last, include one issue per report, and stay responsive afterwards.
The report has a predictable shape: subject, summary, environment, steps, expected, actual, logs. Reuse it.

There is a free companion checklist on our site that sets out all 13 steps with the check to run at each one and the proof that it worked: Kernel Bug Report Checklist.

Frequently asked questions

Where should I send a kernel bug report?
To the maintainers listed for that subsystem in the MAINTAINERS file, with the most specific mailing list and the Linux Kernel Mailing List in Cc. Use scripts/get_maintainer.pl to find them. Most subsystems do not use a bug tracker, so mail is the normal route.

Can I report a bug in my vendor's kernel to upstream maintainers?
In almost all cases, no. A vendor kernel carries patches upstream developers did not write and cannot change. Report it to the vendor, or reproduce the fault on a vanilla mainline kernel first and report that instead.

Why does a tainted kernel matter for my report?
The taint flag records that something happened which can cause unrelated follow-up errors, such as an earlier Oops or an out-of-tree module. A maintainer cannot separate your bug from that noise, so reports from tainted kernels are often set aside.

How long should I wait before following up?
One to five business days is the normal reply window. For an ordinary issue, wait two to three weeks before a friendly reminder. High priority issues, meaning regressions, security issues and really severe problems, warrant a reminder within a week.