Configuration and Debugging of PCIe Endpoint (EP Mode) on OK3588-C Linux 5.10.66 Buildroot
With the rapid development of high-performance embedded applications, the RK3588 processor has become the preferred platform for development boards and industrial applications due to its powerful multi-core computing power and rich peripheral interfaces. In the Linux 5.10.66 Buildroot system based on the Forlinx OK3588-C, the configuration and debugging of the PCIe Endpoint (EP) mode are crucial for achieving high-speed data transmission and device interconnection. This article will provide a detailed introduction on how to complete the kernel configuration, U-Boot settings, and performance testing of the EP device on the RK3588 development board, helping developers quickly set up an efficient and stable PCIe Endpoint environment.
The document testing is based on the RK3588 development board.
- During the testing, both the RC (Root Complex, host device) and the EP (Endpoint, peripheral device) are Linux devices (RK3588 development boards).
- This article focuses on describing the configuration and testing process of the EP device. For detailed configuration information, please refer to the document: Rockchip_Developer_Guide_PCIe_EP_Stardard_Card_CN.pdf.
- Software：Linux 5.10.66 4G+32G
Kernel version: OK3588 linux 5.10.66 version kernel (Android12 Linux ForlinxDesktop20.04 R5 release)
Uboot version: OK3588 protection encryption uboot (Android12 Linux ForlinxDesktop20.04 R4 release)
Hardware connection:
For the hardware connection, only one PCIe 3x4 lane male-to-male extension cable is required.
Software settings
Applying patches
To set the PCIe-EP mode on the 3588, Rockchip officially provides the configuration reference document and source code patches for the PCIe EP mode. The document is located in the docs directory.
Note:
1. The patch file is the basic configuration for the EP card. There is also an update package in the data package. The update package contains update patches based on the base patch, mainly for optimizing the base patch and fixing issues. First, apply the base patch, and then replace the corresponding files in the SDK with the files in the update package directory.
2. Both the patch directory and the source code directory contain necessary modifications. Users can choose to directly apply the patch file or replace the corresponding source code files, either method is acceptable.
(1) kernel
The kernel patch file is located at:
Rockchip_PCIe_EP_Stardard_Card_20231215\patch-kernel5.10\linux_base\Patch directory\kernel
Modify the kernel according to the patch file.
Modify the kernel according to the patch file.
Then replace the files under the kernel in the update package.
（2）uboot
The U-Boot patch file is located at:
\u-boot
After applying the patch, replace the files in the u-boot in the update package.
(3) rkbin
It is a bit cumbersome to use the patch method to modify the files under rkbin. Directly replace the source code files:
Rockchip_PCIe_EP_Stardard_Card_20231215\patch-kernel5.10\linux_base\Source code directory\rkbin
After replacing the files in the folder, replace the files in the update package to the corresponding positions.
The above configurations are required for both the EP card and the RC card.
General kernel configuration
The following configuration items need to be completed during the actual testing process to enable the functions of the EP card. It is recommended to make modifications according to the following configurations. If there are any objections, please refer to the Rockchip official document:
Rockchip_Developer_Guide_PCIe_EP_Stardard_Card_CN.pdf
1. SRNS
In the SRNS mode, the Refclock uses its own internal clock. Set this option so that the RC card and the EP card each use their own 100M clock generators.
uboot
diff --git a/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c b/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c index 7a24a1d..f1ce41a 100755 --- a/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c +++ b/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c @@ -81,7 +81,7 @@ #define PCIe_ATU_CPU_ADDR_HIGH 0x18 /* SRNS: Use Separate refclk(internal clock) instead of from RC */ -// #define PCIe_ENABLE_SRNS_PLL_REFCLK + #define PCIe_ENABLE_SRNS_PLL_REFCLK
kernel
There is a patch file for the SRNS mode in the kernel configuration:
Rockchip_PCIe_EP_Stardard_Card_20231215\patch-kernel5.10\SRNS
You can directly make corrections under the kernel.
2. Hugepage
Hugepage is a way to allocate memory through the linux HugePageSize. The official Rockchip document recommends using the HugePage method for memory allocation to meet the requirement of large-block memory for high-speed PCIe data transmission.
The kernel configuration file defconfig:
diff --git a/arch/arm64/configs/OK3588-Linux_defconfig b/arch/arm64/configs/OK3588-Linux_defconfig index 59b4bf7ca..62a77acf7 100644 --- a/arch/arm64/configs/OK3588-Linux_defconfig +++ b/arch/arm64/configs/OK3588-Linux_defconfig @@ -734,6 +734,7 @@ CONFIG_VIRTIO_NET=y CONFIG_NLMON=y CONFIG_POSIX_MQUEUE=y CONFIG_PCIe_DW_ROCKCHIP_EP=y +CONFIG_HUGETLBFS=y
Device tree file：
diff --git a/arch/arm64/boot/dts/rockchip/OK3588-C-Linux.dts b/arch/arm64/boot/dts/rockchip/OK3588-C-Linux.dts
index 8ea120342..651f81533 100644
--- a/arch/arm64/boot/dts/rockchip/OK3588-C-Linux.dts
+++ b/arch/arm64/boot/dts/rockchip/OK3588-C-Linux.dts
@@ -5,7 +5,8 @@
compatible = "forlinx,ok3588", "rockchip,rk3588";
chosen: chosen {
- bootargs = "earlycon=uart8250,mmio32,0xfeb50000 console=ttyFIQ0 irqchip.gicv3_pseudo_nmi=0 root=PARTUUID=614e0000-0000 rw rootwait";
+ // bootargs = "earlycon=uart8250,mmio32,0xfeb50000 console=ttyFIQ0 irqchip.gicv3_pseudo_nmi=0 root=PARTUUID=614e0000-0000 rw rootwait";
+ bootargs = "earlycon=uart8250,mmio32,0xfeb50000 console=ttyFIQ0 irqchip.gicv3_pseudo_nmi=0 clk_gate.always_on=1pm_domains.always_on=1 root=PARTUUID=614e0000-0000 rw rootwaitdefault_hugepagesz=32M hugepagesz=32M hugepages=16 ";
};
cspmu: cspmu@fd10c000 {
3. BAR
The setup of BAR follows the official rk documentation
uboot
diff --git a/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c b/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c
index 7a24a1d..f1ce41a 100755
--- a/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c
+++ b/arch/arm/mach-rockchip/spl_PCIe_ep_boot.c
@@ -201,14 +201,14 @@ static void PCIe_bar_init(void *dbi_base)
writel(val, dbi_base + 0x7c);
/* Resize BAR0 to support 4M 32bits */
- resbar_base = dbi_base + PCI_RESBAR;
- writel(0x40, resbar_base + 0x4);
+ resbar_base = dbi_base + 0x2e8;
+ writel(0xfffff0, resbar_base + 0x4);
writel(0x2c0, resbar_base + 0x8);
/* BAR2: 64M 64bits */
- writel(0x400, resbar_base + 0x14);
+ writel(0xfffff0, resbar_base + 0x14);
writel(0x6c0, resbar_base + 0x18);
/* BAR4: Fixed for EP wired register, 1M 32bits */
- writel(0x10, resbar_base + 0x24);
+ writel(0xfffff0, resbar_base + 0x24);
writel(0xc0, resbar_base + 0x28);
/* Set flags */
rockchip_PCIe_ep_set_bar_flag(dbi_base, 0, PCI_BASE_ADDRESS_MEM_TYPE_32);
kernel
diff --git a/drivers/pci/controller/dwc/PCIe-dw-ep-rockchip.c b/drivers/pci/controller/dwc/PCIe-dw-ep-rockchip.c
index 1addd6c38..55e7756d4 100755
--- a/drivers/pci/controller/dwc/PCIe-dw-ep-rockchip.c
+++ b/drivers/pci/controller/dwc/PCIe-dw-ep-rockchip.c
@@ -532,18 +532,18 @@ static void rockchip_PCIe_resize_bar(struct rockchip_PCIe *rockchip)
/* Resize BAR0 4M 32bits, BAR2 64M 64bits-pref, BAR4 1MB 32bits */
bar = BAR_0;
- dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0x40);
+ dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0xfffff0);
dw_PCIe_writel_dbi(pci, resbar_base + 0x8 + bar * 0x8, 0x2c0);
rockchip_PCIe_ep_set_bar_flag(rockchip, bar, PCI_BASE_ADDRESS_MEM_TYPE_32);
bar = BAR_2;
- dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0x400);
+ dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0xfffff0);
dw_PCIe_writel_dbi(pci, resbar_base + 0x8 + bar * 0x8, 0x6c0);
rockchip_PCIe_ep_set_bar_flag(rockchip, bar,
PCI_BASE_ADDRESS_MEM_PREFETCH | PCI_BASE_ADDRESS_MEM_TYPE_64);
bar = BAR_4;
- dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0x10);
+ dw_PCIe_writel_dbi(pci, resbar_base + 0x4 + bar * 0x8, 0xfffff0);
dw_PCIe_writel_dbi(pci, resbar_base + 0x8 + bar * 0x8, 0xc0);
rockchip_PCIe_ep_set_bar_flag(rockchip, bar, PCI_BASE_ADDRESS_MEM_TYPE_32)
The above configurations are required for both the EP card and the RC card.
Uboot general settings
Uboot is configured in defconfig
diff --git a/configs/OK3588-Linux_defconfig b/configs/OK3588-Linux_defconfig index cde4d18..e6228b8 100644 --- a/configs/OK3588-Linux_defconfig +++ b/configs/OK3588-Linux_defconfig +CONFIG_SPL_PCIe_EP_SUPPORT=y +CONFIG_SPL_RAM_SUPPORT=y +CONFIG_SPL_RAM_DEVICE=y
The above configurations are required for both the EP card and the RC card.
RC kernel configuration
The configuration on the RC side is simple.Only need to add the configuration in defconfig.
diff --git a/arch/arm64/configs/OK3588-Linux_defconfig b/arch/arm64/configs/OK3588-Linux_defconfig index 3506488cf..26d8f2ae8 100644 --- a/arch/arm64/configs/OK3588-Linux_defconfig +++ b/arch/arm64/configs/OK3588-Linux_defconfig @@ -733,3 +733,8 @@ CONFIG_VETH=y CONFIG_VIRTIO_NET=y CONFIG_NLMON=y CONFIG_POSIX_MQUEUE=y +CONFIG_PCIe_FUNC_RKEP=y
EP kernel configuration
The kernel of the EP (Endpoint) side needs to be configured. Note that if the `CONFIG_STRICT_DEVMEM` option is configured in the kernel, it should be removed.
diff --git a/arch/arm64/configs/OK3588-Linux_defconfig b/arch/arm64/configs/OK3588-Linux_defconfig index 3506488cf..59b4bf7ca 100644 --- a/arch/arm64/configs/OK3588-Linux_defconfig +++ b/arch/arm64/configs/OK3588-Linux_defconfig @@ -733,3 +733,8 @@ CONFIG_VETH=y CONFIG_VIRTIO_NET=y CONFIG_NLMON=y CONFIG_POSIX_MQUEUE=y +CONFIG_PCIe_DW_ROCKCHIP_EP=y
Device tree configuration
diff --git a/arch/arm64/boot/dts/rockchip/OK3588-C-common.dtsi b/arch/arm64/boot/dts/rockchip/OK3588-C-common.dtsi
index 6dc06d53e..1848560bb 100644
--- a/arch/arm64/boot/dts/rockchip/OK3588-C-common.dtsi
+++ b/arch/arm64/boot/dts/rockchip/OK3588-C-common.dtsi
@@ -46,8 +46,15 @@
#address-cells =;
#size-cells =;
ranges;
- dma_trans: dma-trans@3c000000 {
+ /* dma_trans: dma-trans@3c000000 {
reg =;
+ };*/
+
+ bar0_region: bar0-region@3c000000 {
+ reg =;
+ };
+ bar2_region: bar2-region@40000000 {
+ reg =;// # Bar大小配置：0x04000000Bytes
};
/* Reserve 256MB memory for hdmirx-controller@fdee0000 */
@@ -954,8 +961,11 @@
};
&PCIe3x4 {
+ compatible = "rockchip,rk3588-PCIe-std-ep";
reset-gpios = <&gpio4 RK_PB6 GPIO_ACTIVE_HIGH>;
- memory-region = <&dma_trans>;
+// memory-region = <&dma_trans>;
+ memory-region = <&bar0_region>, <&bar2_region>;
+ memory-region-names = "bar0", "bar2";
vPCIe3v3-supply = <&vcc3v3_PCIe30>;
status = "okay";
};
Test
1. Verification
After the configuration in the above steps, the following phenomena should be observed on the development board:
RC Side
PCIe-related nodes can be seen under the /dev directory.
Successful registration and usage of the rkep driver can be queried.
EP Side
PCIe dependent nodes can be seen under dev.
2. Demo Test
There are many test demos in the data package provided by Rockchip. Here, the speed-testing demo is selected:
Rockchip_PCIe_EP_Stardard_Card_20231215\examples\PCIe_speed_test
This demo needs to be compiled under Linux. Copy the compiled executable file to the development board for testing.
According to the information in the README.txt file in the examples directory, the resources required by the demo need to be downloaded.
Considering that the dynamic libraries and resources on which the test application depends are relatively large, a resource directory is created to specifically store the test resources.
Download the resource directory to the examples/ directory via Baidu Netdisk.
Link: https://pan.baidu.com/s/1dyXEXMUfWREZii0kHbkOFw?pwd=u1h7
Extraction code: u1h7
Directory Structure
examples/ └── resource ├── PCIe_camera_test ├── PCIe_speed_test └── PCIe_video_test
After downloading the resources, copy the entire resource package to any directory in the Linux source code. Configure the compilation toolchain in the compilation tool script according to the README.txt document in the speed-testing demo.
The main modifications are as follows:
## build_ep.sh file
#/bin/sh!
EP_CROSS_COMPILE=/home/forlinx/aarch64-buildroot-linux-gnu_sdk-buildroot/bin/aarch64-linux- //Configure the toolchain
# hugepage/rkdrm
EP_MEM_CONFIG=hugepage
make -f ep.mk CROSS_COMPILE=${EP_CROSS_COMPILE} MEM_CONFIG=${EP_MEM_CONFIG} $1
## build_rc.sh file
#/bin/sh!
#RC_ARCH=x86
#RC_CROSS_COMPILE=
RC_ARCH=aarch64
RC_CROSS_COMPILE=/home/forlinx/aarch64-buildroot-linux-gnu_sdk-buildroot/bin/aarch64-linux- //Configure the toolchain
# hugepage/rkep/rkdrm
RC_MEM_CONFIG=hugepage
RC_HOST=${RC_ARCH}-linux
make -f rc.mk CROSS_COMPILE=${RC_CROSS_COMPILE} HOST=${RC_HOST} MEM_CONFIG=${RC_MEM_CONFIG} $1
!!! The path of the above toolchain should be modified according to the actual path in your development environment !!!
Execute the two .sh files respectively in the script directory, and you will get the speed-testing demo executable files for the RC (Root Complex) side and the EP (Endpoint) side.
Copy them to the corresponding boards for speed testing.
The README.txt in the speed-testing demo folder tells you how to use the executable files:
Rockchip_PCIe_EP_Stardard_Card_20231215\examples\PCIe_speed_test
EP side test script running
RC side test script running
Actual speed-testing results on the EP side
Actual speed-testing results on the RC side
Problems
1. The board fails to enter the system
It is found that the board gets stuck during startup. The phenomenon on the terminal interface is as follows:
After waiting for some time, the blue light on the core board flashes abnormally.
This is because the configuration in RK3588MINIALL.ini was modified during the configuration process.
It should be set to all 0s by default.
If the EP card still cannot enter the system after trying the above method, try to connect the RC card.
2. An error occurs when using the speed-testing demo on the RC side
The above error is summarized as a problem that the program encounters when allocating huge-page memory.
At this time, you can check the number of hugepages:
cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
You can see that the number is 0. Set a number for it using echo:
echo 1000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
This can solve the above error.
PS: This configuration also needs to be checked on the EP side.
The above information can also be found in /proc/meminfo:
cat /proc/meminfo | grep Huge
If it is 0, you need to set the number of hugepages.
