OK1126B-S (Linux 6.1.141) Dev Notes: Hardware-Accelerated RTSP AV Streaming Implementation
Source Code Modification
Modify the defconfig file and add the rtsp package.
Add the following configuration to the file buildroot/configs/rockchip_ok1126b-s_defconfig.
BR2_PACKAGE_GST1_RTSP_SERVER=y
Then recompile buildroot, ensuring that the compiled folder contains the following items:
forlinx@ubuntu:~/35xx-nas/1126b/OK1126B-linux-source/buildroot/output/latest/build/gst1-rtsp-server-1.24.11$ ls AUTHORS ChangeLog COPYING.LIB examples gst1-rtsp-server-1.24.11.tar meson.build NEWS RELEASE scripts TODO build COPYING docs gst gst-rtsp-server.doap meson_options.txt README REQUIREMENTS tests
Modification and Compilation of the Test Demo
This modification integrates the test paths and node information of the current board, so the compiled tool can run directly without parameter configuration. The default configuration of this file supports adding parameters to drive different audio and video nodes, and you can decide whether to make modifications as needed.
The storage path of the demo in the source code is: buildroot/output/latest/build/gst1-rtsp-server-1.24.11/examples
There is a file named test-launch.c, which is the demo source file we need for testing.
You can directly modify it to:
#include <gst/gst.h>
#include <gst/rtsp-server/rtsp-server.h>
int main(int argc, char *argv[]) {
GstRTSPServer *server;
GstRTSPMountPoints *mounts;
GstRTSPMediaFactory *factory;
// Initialize GStreamer
gst_init(&argc, &argv);
// Create and configure an RTSP server
server = gst_rtsp_server_new();
g_object_set(server, "service", "8554", NULL); // Specify port
mounts = gst_rtsp_server_get_mount_points(server);
// Create a media factory
factory = gst_rtsp_media_factory_new();
gst_rtsp_media_factory_set_shared(factory, TRUE);
gst_rtsp_media_factory_set_protocols(factory, GST_RTSP_LOWER_TRANS_TCP);
// Optimised pipeline configuration
gst_rtsp_media_factory_set_launch(factory,
"( "
"v4l2src device=/dev/video23 io-mode=4 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! videoscale ! video/x-raw,width=1280,height=720 ! mpph264enc rc-mode=1 bps=2000000 gop=30 ! h264parse ! rtph264pay name=pay0 pt=96 "
"alsasrc device=plughw:0,0 ! queue ! audioconvert ! audioresample ! audio/x-raw,rate=44100,channels=2,format=S16BE ! rtpL16pay name=pay1 pt=97 "
")"
);
//Bind to path /stream
gst_rtsp_mount_points_add_factory(mounts, "/test", factory);
g_object_unref(mounts);
//Start the service
gst_rtsp_server_attach(server, NULL);
g_print("RTSP server start：rtsp://localhost:8554/test\n");
g_main_loop_run(g_main_loop_new(NULL, FALSE));
return 0;
}
After the modification, create a dedicated directory for compiling this demo under another separate path, and move the revised test-launch.c into this newly created directory.
Then configure the cross-compilation environment — be sure to adjust the specific configuration items strictly according to the actual path of your local development environment!! ）
export SDK_DIR=~/35xx-nas/1126b/OK1126B-linux-source export PATH=$SDK_DIR/buildroot/output/rockchip_ok1126b-s/host/bin:$PATH export PKG_CONFIG_SYSROOT_DIR=$SDK_DIR/buildroot/output/rockchip_ok1126b-s/host/aarch64-buildroot-linux-gnu/sysroot export PKG_CONFIG_PATH=$PKG_CONFIG_SYSROOT_DIR/usr/lib/pkgconfig ### Compilation aarch64-buildroot-linux-gnu-gcc test-launch.c -o test-launch $(pkg-config --cflags --libs gstreamer-1.0 gstreamer-rtsp-server-1.0)
After the compilation, a binary file named test-launch will be generated. Copy this file to the target board, and grant executable permission to it.
Test
The devices used for the test are the OK1126B board and the VLC media player on a PC.
Before starting the test, make sure that both devices are connected to the same local area network (LAN).
root@OK1126B-buildroot:~# ifconfig eth0 Link encap:Ethernet HWaddr 1A:75:63:08:53:22 inet addr:172.20.2.151 Bcast:172.20.2.255 Mask:255.255.255.0 inet6 addr: fe80::1875:63ff:fe08:5322/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:36928 errors:0 dropped:1513 overruns:0 frame:0 TX packets:22624 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2894353 (2.7 MiB) TX bytes:25825279 (24.6 MiB) Interrupt:79 root@OK1126B-buildroot:~# ping 172.20.2.51 PING 172.20.2.51 (172.20.2.51) 56(84) bytes of data. 64 bytes from 172.20.2.51: icmp_seq=1 ttl=128 time=0.535 ms 64 bytes from 172.20.2.51: icmp_seq=2 ttl=128 time=0.473 ms 64 bytes from 172.20.2.51: icmp_seq=3 ttl=128 time=0.464 ms --- 172.20.2.51 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2043ms rtt min/avg/max/mdev = 0.464/0.490/0.535/0.031 ms
After that, you can run the pre-compiled test demo to start the verification.
In this test, the video input source is the ov13850 MIPI camera, and the audio input uses the onboard MIC.
The detailed test procedure is as follows:
### Board end operation root@OK1126B-buildroot:~# GST_DEBUG=3 ./test-launch RTSP server start?rtsp://localhost:8554/test mpp[1904]: mpp_info: mpp version: unknown mpp version for missing VCS info mpp[1904]: mpp_info: mpp version: unknown mpp version for missing VCS info mpp[1904]: mpp_info: mpp version: unknown mpp version for missing VCS info mpp[1904]: mpp: unable to create enc vp8 for soc rv1126b unsupported mpp[1904]: mpp_info: mpp version: unknown mpp version for missing VCS info Using mplane plugin for capture 0:00:00.555670165 1904 0x7f9c000d90 FIXME default gstutils.c:4089:gst_element_decorate_stream_id_internal:<alsasrc0> Creating random stream-id, consider implementing a deterministic way of creating a stream-id 0:00:00.555929178 1904 0x7f9c000f80 WARN rtspstream rtsp-stream.c:5381:rtp_pad_blocking: Passing event. mpp[1904]: mpp_info: mpp version: unknown mpp version for missing VCS info mpp[1904]: mpp: Only rk3588's h264/265/jpeg and rk3576's h264/265 encoder can use frame parallel 0:00:00.557817233 1904 0x7f9c001380 WARN v4l2 gstv4l2object.c:4984:gst_v4l2_object_probe_caps:<v4l2src0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument 0:00:00.560072222 1904 0x7f9c000d90 WARN alsa pcm_hw.c:1429:snd_pcm_hw_get_chmap: alsalib error: Cannot read Channel Map ctl: No such file or directory mpp[1904]: mpp_enc: MPP_ENC_SET_RC_CFG bps 2000000 [1875000 : 2125000] fps [30:30] gop 30 mpp[1904]: mpp_enc: mode cbr bps [1875000:2000000:2125000] fps fix [30/1] -> fix [30/1] gop i [30] v [0] 0:00:00.562769693 1904 0x7f9c001170 WARN rtspstream rtsp-stream.c:5381:rtp_pad_blocking: Passing event. 0:00:00.563530106 1904 0x7f9c000f80 WARN rtspstream rtsp-stream.c:5381:rtp_pad_blocking: Passing event. 0:00:00.563812162 1904 0x7f9c001380 WARN v4l2 gstv4l2object.c:4768:gst_v4l2_object_get_crop_rect:<v4l2src0:src> Failed to get default crop rectangle with VIDIOC_G_SELECTION: Invalid argument 0:00:00.563912209 1904 0x7f9c001380 WARN v4l2 gstv4l2object.c:3356:gst_v4l2_object_reset_compose_region:<v4l2src0:src> Failed to get default compose rectangle with VIDIOC_G_SELECTION: Invalid argument 0:00:00.574747468 1904 0x7f9c000f80 WARN rtspstream rtsp-stream.c:5381:rtp_pad_blocking: Passing event. 0:00:00.574808138 1904 0x7f9c000f80 WARN rtpsession gstrtpsession.c:2484:gst_rtp_session_chain_send_rtp_common:<rtpsession1> Can't determine running time for this packet without knowing configured latency 0:00:00.653885533 1904 0x7f9c001380 WARN v4l2bufferpool gstv4l2bufferpool.c:1345:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool0:src> Driver should never set v4l2_buffer.field to ANY current rga_api version 1.10.5_[4]. The called RockchipRga API is deprecated and will be removed in a future release, currently using the legacy API for compatibility. Please refer to the latest IM2D API documentation to update your implementation. 0:00:00.669052679 1904 0x7f9c001590 WARN rtspstream rtsp-stream.c:5381:rtp_pad_blocking: Passing event. 0:00:00.669681253 1904 0x7f9c001590 WARN rtspstream rtsp-stream.c:5381:rtp_pad_blocking: Passing event. 0:00:00.669855970 1904 0x7f9c001590 WARN rtspstream rtsp-stream.c:5381:rtp_pad_blocking: Passing event. 0:00:00.669924223 1904 0x7f9c001590 WARN rtpsession gstrtpsession.c:2484:gst_rtp_session_chain_send_rtp_common:<rtpsession0> Can't determine running time for this packet without knowing configured latency 0:00:00.670959109 1904 0x7f9c000b70 FIXME rtspmedia rtsp-media.c:4892:gst_rtsp_media_suspend: suspend for dynamic pipelines needs fixing 0:00:00.672962378 1904 0x7f9c000b70 FIXME rtspmedia rtsp-media.c:4892:gst_rtsp_media_suspend: suspend for dynamic pipelines needs fixing 0:00:00.673009047 1904 0x7f9c000b70 WARN rtspmedia rtsp-media.c:4932:gst_rtsp_media_suspend: media 0x7f94064b00 was not prepared 0:00:00.673235975 1904 0x7f9c000b70 WARN rtspclient rtsp-client.c:2368:parse_transport: unsupported transport RTP/AVP;unicast;client_port=49724-49725 0:00:00.673269518 1904 0x7f9c000b70 ERROR rtspclient rtsp-client.c:3187:handle_setup_request: client 0x55b6340af0: unsupported transports 0:00:00.674477372 1904 0x7f9c000b70 FIXME rtspmedia rtsp-media.c:4892:gst_rtsp_media_suspend: suspend for dynamic pipelines needs fixing 0:00:00.674518207 1904 0x7f9c000b70 WARN rtspmedia rtsp-media.c:4932:gst_rtsp_media_suspend: media 0x7f94064b00 was not prepared 0:00:00.680123450 1904 0x7f9c001780 FIXME default gstutils.c:4089:gst_element_decorate_stream_id_internal:<appsrc0> Creating random stream-id, consider implementing a deterministic way of creating a stream-id 0:00:00.680226414 1904 0x7f9c001780 WARN GST_PADS gstpad.c:4414:gst_pad_peer_query:<appsrc0:src> could not send sticky events 0:00:00.683157396 1904 0x7f9c001970 FIXME default gstutils.c:4089:gst_element_decorate_stream_id_internal:<appsrc1> Creating random stream-id, consider implementing a deterministic way of creating a stream-id 0:00:00.684377792 1904 0x7f9c000b70 FIXME rtspmedia rtsp-media.c:3147:gst_rtsp_media_seek_trickmode:<GstRTSPMedia@0x7f94064b00> Handle going back to 0 for none live not seekable streams.
PC end execution：
Launch the VLC software, navigate to Media > Open Network Stream, select the Network tab, then enter the network URL — the port and stream path are pre-defined and fixed in the demo.
Note that the option displayed in the playback dropdown may not show ''Play'' by default: open the dropdown menu, select the Play option, then confirm your selection.
At this point, the entire setup process is completed. You will now be able to view the camera feed in VLC. If you connect a pair of headphones to the PC, speak or make sounds near the board''s onboard MIC, and you will hear the corresponding audio played through the headphones.
Phase 1: Deep Processing on the Board Side (Capture & Encode)
Peripheral Capture: * Video: The camera sensor detects light, which is transmitted via the MIPI interface to the RK chip; the V4L2 driver then outputs the raw NV12-format image via the /dev/video23 node.
AUDIO: The microphone captures sound waves, which are converted into electrical signals via the ADC, and the ALSA software interface plughw:0,0 outputs 44.1kHz raw PCM digital audio.
Hardware Encoding and Processing
Raw unprocessed data is far too large to transmit over Ethernet.Video frames are sent to Rockchip''s dedicated MPP hardware encoder (mpph264enc), which instantly compresses them into a compact H.264 bitstream.
Audio data goes through resampling and endian conversion (to S16BE format), and is prepared for transmission as L16 raw bitstream.
Payload Packetization：Large raw data cannot be transmitted over the network as a single bulk file. The rtph264pay and rtpL16pay elements split the H.264 video stream and audio stream into small fragments, encapsulate them into RTP packets, and attach timestamps to ensure subsequent audio-video synchronization.
Phase 2: Transmission Over Ethernet
The timestamped RTP packets are sent at high frequency over the Ethernet cable to the PC at IP address 172.20.2.51.
Phase 3: Reverse Decoding and Rendering on the PC Side (Decode & Render)
Reception and Depayload: VLC receives RTP packets, strips off network headers, and reassembles fragments into complete H.264 video streams and L16 audio streams.
Decode: The * PC's CPU/GPU invokes the video decoder to restore H.264 into individual displayable image frames.
Since the audio is a lossless raw bitstream, it requires barely any decompression processing.
Synchronization and Rendering: This is the most critical step of the whole workflow. VLC will verify the timestamps attached to the audio and video streams. If it detects that the video is playing faster than the audio, it will hold back the video frames for a few milliseconds for timing adjustment. Once aligned, the video signal is sent to the monitor and the audio signal to the speakers.
