Linux Kernel Meet

I recently received an DM from a kernel developer working on upstreaming an ADC driver. They hit a classic and frustrating wall: The driver worked perfectly on v6.1, but after porting it to the latest mainline (v6.18-rc5), the boot logs stopped dead at "Starting Kernel...". No panic message, no earlycon, just silence.

This is a scenario many of us faced when we first started kernel development. Here is the advice I shared, which might be helpful for anyone dealing with major kernel upgrades.

1. Don't try to jump 17 floors at once (Incremental Updates)

Moving directly from v6.1 to v6.18 is like trying to jump from the 1st floor to the 18th without an elevator. It’s a recipe for broken legs (and broken builds).

The Analogy: Treat kernel upgrades like crossing a river with stepping stones.
The Fix: Instead of the latest RC, try porting to closer Long Term Support (LTS) versions first (e.g., v6.1 -> v6.6 -> v6.10).
The Value: Making your driver work on these intermediate stable kernels is not just "busy work"—it is a valuable contribution and the best way to understand what changed and when.

2. "git bisect" is your best friend (but narrow the search first)

When a regression happens, "git bisect" is the standard tool to find the culprit commit. However, the gap between v6.1 and v6.18 contains tens of thousands of commits. Finding a needle in that haystack is painful. By following step (Incremental Updates), you can narrow the range (e.g., "It works on v6.10 but breaks on v6.11") before running bisect. This makes the process much faster and more manageable.

git bisect document: https://git-scm.com/docs/git-bisect
"git bisect" results in the kernel mailing lists: https://lore.kernel.org/all/?q=q%3A"git+bisect"

3. Leave "Breadcrumbs" in the dark (Manual Tracing)

If the system hangs even before initializes, standard debugging tools often won't help. The kernel is crashing before it has a voice.

The Analogy: Like Hansel and Gretel, you need to leave breadcrumbs to see how far you got.
The Fix: Go "old school." Manually insert print statements (pr_info() or printk()) directly into the early initialization code, such as start_kernel() in init/main.c: “I am here 1”, “I am here 2”...
It looks primitive, but seeing where the printing stops will tell you exactly which function caused the panic.

By configuring the earlycon parameter, the kernel can output messages during the initial boot phase, before standard consoles are initialized. This allows us to capture the kernel call trace.

See "Detailed Explanation of setup_earlycon" by David Zhu

https://www.linkedin.com/pulse/detailed-explanation-setupearlycon-david-zhu-lai0c/

The Linux kernel is massive, but if you break the problem down into smaller, manageable steps, no bug is unfixable. Happy hacking!

Yunseong Kim

Hi @Jinseok, Thank you for the great sharing!

Yunseong Kim

The Linux kernel build system (Kbuild) detects changed files using a combination of make's timestamp comparison and a sophisticated Dependency Tracking mechanism.

As you observed, changing a single .c file finishes quickly, whereas modifying a header (.h) file often triggers a massive rebuild. This happens because of the structure of the Dependency Graph.
Here is the technical breakdown of how this works:

Core Detection Mechanism: Hidden .cmd Files
When the kernel is compiled, Kbuild doesn't just generate the .o (object) file; it also creates a hidden file that records a list of every single file referenced during the creation of that object.

Example: Let's say you are compiling drivers/net/ethernet/intel/e1000/e1000_main.c.
The build system generates e1000_main.o and simultaneously creates a hidden file named .e1000_main.o.cmd in the same directory.
If you open this .cmd file, you will see something like this (it is very long):

cmd_drivers/net/ethernet/intel/e1000/e1000_main.o := gcc ... (compiler options) ... \
  source_drivers/net/ethernet/intel/e1000/e1000_main.c \
  deps_drivers/net/ethernet/intel/e1000/e1000_main.o := \
    drivers/net/ethernet/intel/e1000/e1000_main.c \
    include/linux/module.h \
    include/linux/types.h \
    include/linux/init.h \
    ... (hundreds of other header files) ...

The next time you run make, the process is as follows:

It checks if e1000_main.o exists.
It reads the .e1000_main.o.cmd file.
If the modification time (timestamp) of any source or header file listed there is newer than the e1000_main.o file, make determines that the object is "outdated" and recompiles it.

Why Does Modifying a Header Trigger a Full Rebuild?
This is due to the difference in the Scope of Reference.
A. When modifying a .c file (Leaf Node Change)

my_driver.c is typically not included by other files. It sits at the edge (leaf) of the dependency tree.
Therefore, if my_driver.c changes, only my_driver.o needs to be rebuilt.
B. When modifying a .h file (Root/Branch Node Change)
Take a core header like include/linux/sched.h (related to process scheduling) as an example.
This header is #included by almost every subsystem in the kernel (filesystems, memory management, drivers, etc.).
Consequently, include/linux/sched.h appears in the dependency lists within thousands of .cmd files.
The moment this header is touched, make sees that "thousands of .o files that depend on this header are outdated," and triggers a rebuild for all of them.

Note: In C, recompilation upon header modification is strictly necessary. If structure (struct) member offsets change or macro (#define) values are updated, the binary code utilizing them must be regenerated to calculate the new offsets/values correctly.

Technical Implementation Tool: fixdep
To optimize this process, the Linux kernel build system uses a tool called fixdep.

When compiling with GCC, the -Wp,-MMD,file.d option is used. This tells the compiler to output a list of all header files read during the Preprocessing stage.
The kernel's scripts/basic/fixdep program parses this list and converts it into the .cmd file format (Make rules) shown above.
It also tracks changes to command-line options (CFLAGS). If the compiler flags change, fixdep ensures a rebuild occurs even if the source files haven't changed.
Summary
How does it know?: Every .o file has a hidden .cmd file that lists "every file used to build me (including headers)."
Why do headers take so long?: Because that single header file is listed in the .cmd dependency lists of thousands of object files.
Why are .c files fast?: They are rarely dependencies for other files, so only the file itself needs recompilation.
Tip: How to build specific modules quickly?
Running make from the kernel root checks the entire dependency tree, which can be slow. You can build specific directories to save time:

# Build only the driver in this specific directory
make M=drivers/net/ethernet/intel/e1000/

However, if you have modified a core header (include/linux/*.h), a full rebuild is unavoidable. Skipping it would likely result in a Symbol Version (CRC) mismatch, preventing the module from loading.

1. File Change Check After Build

Checking Config (m vs * ): If it is a module (m), you only need to run make modules_install and reload it. If it is built-in (* or y), you must reinstall the kernel image (bzImage) and reboot.
Checking Timestamps: Needs refinement. The make utility handles timestamp comparisons automatically.
Comparing md5sum: The .o or .ko files in your build directory will have different hash values every time you build them (due to build paths, timestamps, etc.). Furthermore, installed modules undergo stripping (removal of debug symbols), making their hash different from the original build artifact.

2. Verifying Steps

Step 1: Verify Config Status (Improve Precision)

Using grep is more reliable than visually checking menuconfig.

Method: Check the .config file in your working directory.

grep "CONFIG_MY_DRIVER" .config

CONFIG_...=m: It is a module. Just replace the .ko file (No reboot needed).
CONFIG_...=y: It is built-in. You must flash the new kernel image and reboot.

Step 2: Verify Changes (Key Correction)

Comparing timestamps or md5sum hashes is inaccurate. Instead, verifying strings inside the binary or using Modinfo is the most definitive method.

Recommended Method A: Use strings (Most Reliable)
Add a unique, identifiable string (like a log message) to your code, then check if that string exists in the binary.

Add code: printk(KERN_INFO "MyPatch_v1 checking...\n");
Build, then verify:

strings drivers/my/driver.ko | grep "MyPatch_v1"

If you see the string, your changes are 100% applied.

**Recommended Method B: Check modinfo**
If you updated the module version, check it with modinfo.

modinfo drivers/my/driver.ko

Step 3: Note on Comparing with Installed Modules (Replacing hash sum command)

If you try to compare the "installed module (/lib/modules/.../my.ko)" with the "just built module (my.ko)" using like md5sum, it will fail 99% of the time.

Reason: During make modules_install, debug symbols are stripped, and module signatures are added.
Alternative: Compare file sizes (the installed one will be much smaller) or use the strings command mentioned above to peek inside the installed module.

3. Summary

After modifying the kernel code, check the .config file to determine if the feature is a module (m) or built-in (*, y).

If Module (m): Recompile only the module and reinstall it (rmmod -> insmod).

If Built-in (y): You must reinstall the entire kernel image and reboot the system.

How to Verify Changes:
Simple timestamp or hash (md5sum) comparisons can be inaccurate due to symbol stripping or signing during installation. Instead, I recommend using the strings [module].ko | grep [unique_string] command to verify that your specific code changes are actually present in the binary.

Yunseong Kim

Pre-study before kernel contribution

Q&A: Getting Started with AppArmor Contribution

Q. I’m very interested in AppArmor and would like to contribute. How should I prepare, and what is the best way to start contributing to Linux security?

A. I recommend starting by analyzing the interaction between user space and the kernel. Specifically, you should examine the AppArmor-related options in /proc and /sys, and profile the user space applications that control AppArmor using tools like trace-cmd. This will help you understand the code flow and how policies are enforced.

To give you more context, here is a breakdown of the key components you should look into:

1. Understanding the Interfaces: /sys and /proc
AppArmor relies heavily on the SecurityFS interface to interact with the kernel.

# With root privilege, For example:
$ trace-cmd record -F -p function_graph cat /sys/kernel/security/apparmor/profiles
$ trace-cmd report > cat-apparmor.log

/sys/kernel/security/apparmor: This is the primary control interface. It is where the kernel exports information about the loaded policies, features, and the interface for loading or replacing profiles. Exploring this directory helps you understand what capabilities the running kernel supports.
/proc/[pid]/attr/current: This interface shows the AppArmor profile currently applied to a specific process. Monitoring this file is essential for debugging and verifying that process confinement is working as expected.

2. Analyzing User Space Tools
Before diving deep into the kernel code, it is crucial to understand how policies are generated and passed to the kernel. The core component here is the AppArmor Parser.

apparmor_parser: This is the user space program responsible for compiling human-readable text profiles into a binary format that the kernel can digest. By tracing this application with trace-cmd, you can see exactly how it invokes the kernel interfaces to load policies.

3. Source Code Repository
You can find the source code for the AppArmor user space tools, including the parser and libraries, in their official repository. Studying this code is the best way to understand the mechanism behind policy loading.

AppArmor User Space Repository: https://gitlab.com/apparmor/apparmor
AppArmor Kernel Source Code: https://elixir.bootlin.com/linux/v6.18.2/source/security/apparmor/apparmorfs.c#L2311

Conclusion
Lastly, keep in mind that this workflow is not specific to AppArmor. Analyzing the /sys and /proc interfaces and profiling the interaction between user space and the kernel is a universally effective approach. It serves as an excellent starting point for analyzing and contributing to any Linux kernel subsystem.

P.S.
profiles_open() → seq_read() (call p_start/p_next/p_show/p_stop)

// security/apparmor/apparmorfs.c

static const struct file_operations aa_sfs_profiles_fops = {
    .open = profiles_open,
    .read = seq_read,
    .llseek = seq_lseek,
    .release = profiles_release,
};

profiles_open(): seq_open() init seq_file.
seq_read() - VFS read seq_file.
aa_sfs_profiles_op's callback
- p_start() :Start.
- p_next() : Next profile.
- p_show() : Print profile.
- p_stop() : End
profiles_release() : Close the file.

Yunseong Kim

Oh, great! As for protection view, it's deeply connected to the work I did at my previous job. I'm constantly researching and organizing this topic. I'll cover it in a series.

Yunseong Kim

LF Live Mentorship

https://events.linuxfoundation.org/lf-live-mentorship-series/

Adrian Huang's Linux Kernel Core Big Picture Slides

https://www.slideshare.net/AdrianHuang/presentations

University of Warsaw, Institute of Informatics - Janina Mincer-Daszkiewicz Debian 11 (Kernel 6.12.6) based AOS Course

https://students.mimuw.edu.pl/ZSO/index_en.html

KDLP (Redhat)

https://kdlp.underground.software/course/index.html

Kernel Newbies

Moon Kernel Core Analysis

http://jake.dothome.co.kr/

Linux Kernel Teaching

https://linux-kernel-labs.github.io/

LWN Conference index

https://lwn.net/Archives/ConferenceIndex/

Yocto kernel patch

https://docs.yoctoproject.org/kernel-dev/common.html#modifying-an-existing-recipe

Linux Kernel Exploit

https://github.com/xairy/linux-kernel-exploitation

Linux Kernel CTF

https://github.com/google/security-research/tree/master/pocs/linux/kernelctf

LDD 3rd on v5.10

https://github.com/d0u9/Linux-Device-Driver

Embetronicx Linux Device Driver Tutorial

Kernel Glossary

https://0xff07.github.io/kernel-glossary/

Awesome-kerneldev

https://github.com/retpolanne/awesome-kerneldev

First contribution to Linux kernel

Bangalore Linux Kernel Meetup

https://kernelmeetup.wordpress.com/

Yunseong Kim

Welcome to your brand new NodeBB forum!

This is what a topic and post looks like. As an administrator, you can edit the post's title and content.
To customise your forum, go to the Administrator Control Panel. You can modify all aspects of your forum there, including installation of third-party plugins.

Linux Kernel Meet

Yunseong Kim

Posts

DPDK 2025-eu

Playlist

Index

DPDK 2024-us

Playlist

Index

DPDK 2024-asia

Playlist

Index

DPDK 2023

Playlist

Index

DPDK 2022

Playlist

Index

DPDK 2021-us

Playlist

Index

DPDK 2021-asia

Playlist

Index

DPDK 2020

Playlist

Index

DPDK 2019-us

Playlist

Index

DPDK 2019-india

Playlist

Index

DPDK 2019-eu

Playlist

Index

DPDK 2019-china

Playlist

Index

DPDK 2018-us

Playlist

Index

DPDK 2018-india

Playlist

Index

DPDK 2018-eu

Playlist

Index

DPDK 2018-asia

Playlist

Index

DPDK 2017-us

Playlist

Index

DPDK 2017-eu

Playlist

Index

DPDK 2017-asia

Playlist

Index

DPDK 2016-us

Playlist

Index

DPDK 2016-eu

Playlist

Index

DPDK 2016-asia

Playlist

Index

DPDK 2015-us

Playlist

Index

DPDK 2015-eu

Playlist

Index

LSF MM eBPF 2024

Playlist

Index

LSF MM eBPF 2023

Playlist