In industrial control, motion control, machine vision, edge data acquisition, and other application scenarios, the system must not only run but also operate on time. In scenarios that require high determinism in response, standard Linux often struggles to meet strict real-time requirements related to scheduling latency and interrupt response. As a result, enhancing real-time performance on Linux platforms has become a key focus for many developers.

This article provides a comprehensive overview of the acquisition, porting, compilation adaptation, and real-time verification of the PREEMPT_RT real-time patch on the OK3588 Buildroot platform, along with actual testing processes. Real-time performance before and after applying the patch is compared using cyclictest. The results indicate that the maximum scheduling latency of the system is significantly improved in both idle and stress scenarios.

Why Do We Need PREEMPT_RT?

The standard Linux kernel is not specifically designed for strong real-time scenarios. In the traditional Linux kernel, task preemption points are limited, meaning that certain code paths cannot be interrupted by higher-priority tasks during execution. This is particularly true for critical sections, such as lock management and interrupt handling. As a result, this can lead to scheduling latency with a degree of unpredictability. For general consumer applications, this unpredictability is usually not a significant issue. However, in industrial applications that require stable response cycles and minimal jitter, this inconsistency can become a bottleneck for the system.

The primary goal of PREEMPT_RT is to modify the standard Linux kernel to improve kernel preemptibility as much as possible. This involves transforming interrupt handling, lock mechanisms, and other code paths to better align with real-time requirements. For instance, many interrupt handling processes that originally ran in non-preemptible contexts are converted into schedulable kernel threads. This change allows for finer-grained priority management and minimizes the impact of interrupt blocking on real-time tasks. According to the documentation, these transformations can significantly reduce task response latency, resulting in markedly improved real-time performance compared to the standard kernel.

How to Obtain the Real-Time Patch for the OK3588 Platform？

For the OK3588 platform, the SDK provides the PREEMPT_RT patch corresponding to the relevant version.

ls docs/rk3588/Patches/Real-Time-Performance/PREEMPT_RT/kernel-6.1/kernel-6.1.118/
0001-patch-6.1.99-rt36-on-rockckip-base-5c295c763974.patch
0002-sched-isolation-remove-HK_FLAG_TICK-for-nohz_full-fo.patch
0003-mm-Kconfig-remove-selection-of-MIGRATION-for-CMA-to-.patch
0004-ARM-configs-add-rockchip_rt.config-for-PREEMPT_RT.patch
0005-arm64-configs-optimize-latency-for-PREEMPT_RT.patch
0006-phy-rockchip-inno-usb2-Fix-DEBUG_LOCKS_WARN_ON-in-ch.patch

The path includes several patch files related to real-time performance, such as the foundational RT patch, scheduler isolation optimizations, memory configuration adjustments, additional real-time kernel configurations, and USB-related fixes, among others.

It is important to note that applying PREEMPT_RT is not a simple task of "apply one patch and you're done." Since the preemption points of the real-time patch are distributed throughout nearly the entire kernel, chip manufacturers often need to adjust some underlying drivers accordingly. Therefore, from an engineering perspective, it is more reliable and easier to implement patch sets that have already been curated by chip manufacturers or platform providers, rather than using generic upstream RT patches directly.

How to Port the Real-Time Patch to the Buildroot Kernel？

According to the workflow outlined in the documentation, porting the real-time patch for the OK3588 6.1.118

Batch Application of Patches in the Kernel Directory

First, navigate to the kernel directory and use the patch command to import the real-time patch files provided in the SDK.

for i in ../docs/rk3588/Patches/Real-Time-Performance/\
PREEMPT_RT/kernel-6.1/kernel-6.1.118/*;\
do patch -p1 -i $i; done

Supplementing and Saving the Corresponding Kernel Configuration Items

After entering the patch, you also need to add the corresponding CONFIG item.

# Must be executed, otherwise there will be problems with the compilation environment.
export ARCH=arm64
make menuconfig
make savedefconfig
# You need to select the defconfig corresponding to the SDK
cp defconfig arch/arm64/configs/OK3588-C-linux_defconfig

Operation on the menu is performed by pressing the ''/'' key to search for the PREEMPT option, then pressing 1 to enter it.

Navigate further into the submenu, select the highest real-time priority level, and proceed to compile.

Return to the Top-level SDK Directory and Perform a Full Compilation

After configuration is complete, return to the top-level SDK directory and execute ./build.sh to compile the entire system.

tips:

Encountering errors during compilation is common, as different SDKs incorporate various drivers that may not be adapted to the modifications introduced by the real-time patch.

For instance, by examining the changes made to specific functions within the real-time patch, clues about the issue can often be identified.

cd docs/rk3588/Patches/Real-Time-Performance/PREEMPT_RT/kernel-6.1/kernel-6.1.118/
grep -nr "u64_stats_fetch_begin"
cd -

As observed in the real-time patch, the function `u64_stats_fetch_begin_irq` has been completely replaced by `u64_stats_fetch_begin`. By referencing this modification from the real-time patch and making the corresponding adjustments to the driver, the compilation was successful, allowing the firmware to be flashed for testing.

Real-Time Performance Testing

Download the compressed package and place it on the development board. Execute the command `tar -xvf rt-tests-2.2.tar -C /` to extract the contents into the root directory. The `cyclictest` tool will then be available in `/usr/bin/`.

Testing Without the Real-Time Patch Applied

cyclictest -c 0 -m -t 8 -p 99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.00 0.02 0.00 1/208 3200
T: 0 ( 2749) P:99 I:1000 C: 94901 Min: 2 Act: 3 Avg: 2 Max: 43
T: 1 ( 2750) P:99 I:1500 C: 63265 Min: 2 Act: 2 Avg: 2 Max: 31
T: 2 ( 2751) P:99 I:2000 C: 47446 Min: 2 Act: 3 Avg: 2 Max: 36
T: 3 ( 2752) P:99 I:2500 C: 37956 Min: 2 Act: 3 Avg: 3 Max: 39
T: 4 ( 2753) P:99 I:3000 C: 31628 Min: 2 Act: 3 Avg: 2 Max: 42
T: 5 ( 2754) P:99 I:3500 C: 27109 Min: 2 Act: 2 Avg: 2 Max: 16
T: 6 ( 2755) P:99 I:4000 C: 23720 Min: 2 Act: 2 Avg: 2 Max: 69
T: 7 ( 2756) P:99 I:4500 C: 21084 Min: 2 Act: 2 Avg: 2 Max: 28

Testing With the Real-Time Patch Applied

No-load Test

root@OK3588-C-buildroot:/# cyclictest -c 0 -m -t 8 -p 99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 2.70 0.93 0.34 1/373 2203
T: 0 ( 1961) P:99 I:1000 C: 46702 Min: 0 Act: 0 Avg: 0 Max: 4
T: 1 ( 1963) P:99 I:1500 C: 31133 Min: 0 Act: 1 Avg: 0 Max: 6
T: 2 ( 1964) P:99 I:2000 C: 23348 Min: 0 Act: 1 Avg: 0 Max: 3
T: 3 ( 1965) P:99 I:2500 C: 18677 Min: 0 Act: 1 Avg: 0 Max: 3
T: 4 ( 1966) P:99 I:3000 C: 15563 Min: 0 Act: 0 Avg: 0 Max: 1
T: 5 ( 1967) P:99 I:3500 C: 13339 Min: 0 Act: 0 Avg: 0 Max: 1
T: 6 ( 1968) P:99 I:4000 C: 11670 Min: 0 Act: 0 Avg: 0 Max: 1
T: 7 ( 1969) P:99 I:4500 C: 10373 Min: 0 Act: 0 Avg: 0 Max: 1

Test Performance Mode

echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy4/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy6/scaling_governor
cyclictest -c 0 -m -t 8 -p 99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 2.03 2.76 2.40 2/345 11678
T: 0 (11652) P:99 I:1000 C: 3968 Min: 0 Act: 1 Avg: 0 Max: 2
T: 1 (11653) P:99 I:1500 C: 2641 Min: 0 Act: 0 Avg: 0 Max: 6
T: 2 (11654) P:99 I:2000 C: 1977 Min: 0 Act: 1 Avg: 0 Max: 2
T: 3 (11655) P:99 I:2500 C: 1581 Min: 0 Act: 1 Avg: 0 Max: 2
T: 4 (11656) P:99 I:3000 C: 1315 Min: 0 Act: 0 Avg: 0 Max: 1
T: 5 (11657) P:99 I:3500 C: 1127 Min: 0 Act: 0 Avg: 0 Max: 1
T: 6 (11658) P:99 I:4000 C: 986 Min: 0 Act: 0 Avg: 0 Max: 1
T: 7 (11659) P:99 I:4500 C: 875 Min: 0 Act: 0 Avg: 0 Max: 1

Pressure Test

stress-ng -c 8 --io 2 --vm 1 --vm-bytes 1024M --timeout 1000000s &
cyclictest -c 0 -m -t 8 -p 99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 9.48 3.99 2.84 11/365 13327
T: 0 (13232) P:99 I:1000 C: 18287 Min: 0 Act: 1 Avg: 1 Max: 7
T: 1 (13233) P:99 I:1500 C: 12192 Min: 0 Act: 1 Avg: 1 Max: 8
T: 2 (13234) P:99 I:2000 C: 9142 Min: 0 Act: 1 Avg: 1 Max: 3
T: 3 (13235) P:99 I:2500 C: 7310 Min: 0 Act: 1 Avg: 1 Max: 4
T: 4 (13236) P:99 I:3000 C: 6088 Min: 0 Act: 0 Avg: 0 Max: 6
T: 5 (13237) P:99 I:3500 C: 5219 Min: 0 Act: 0 Avg: 0 Max: 3
T: 6 (13238) P:99 I:4000 C: 4565 Min: 0 Act: 1 Avg: 0 Max: 3
T: 7 (13239) P:99 I:4500 C: 4056 Min: 0 Act: 0 Avg: 0 Max: 4

Summary

This test based on the OK3588 6.1.118 Buildroot platform demonstrates that the PREEMPT_RT real-time patch can not only be smoothly integrated into the existing SDK but also significantly improve system scheduling latency performance under idle, performance mode, and high-stress scenarios. Particularly from the cyclictest results, the maximum latency after applying the patch shows a marked decrease compared to the pre-patch state, effectively enhancing the system’s real-time determinism.

For Linux projects requiring stronger real-time capabilities, this approach holds substantial engineering and practical value.It is not merely about transforming Linux into ''another RTOS.'' Instead, it moves Linux closer to real-time scenarios while preserving its ecosystem and driver capabilities, offering a more viable technical pathway for industrial-grade applications.

---