In edge computing and industrial IoT scenarios, deploying large language models at the field site is becoming a clear trend. Compared to cloud inference, local deployment ensures physical data isolation, eliminates WAN latency and jitter, and avoids ongoing token subscription costs under high concurrency.

Limited by the computing power and memory bottlenecks of edge hardware, traditional on-device deployments are often confined to 7B-scale models, struggling to reliably handle long-context RAG, structured code generation, and multi-step logical reasoning. To address this, this solution utilizes the high-performance OK3588-C main control board and cascades four RK1828 AI accelerator cards via the PCIe bus. Within a total power budget of approximately 40W for the accelerator cluster, it successfully enables full-featured local inference of medium-to-large models such as Qwen3.8-27B and Gemma-4-31B-it.

Why Are Enterprises Seeking the Optimal Solution for Local Large Models?

Before diving into the technical details, let’s first address the core business pain points.

For highly regulated industries such as industrial manufacturing, rail transportation, and finance, migrating core business data to the cloud consistently faces three major challenges:

• Compliance and Security: Keeping sensitive data “within the domain” is a strict requirement. Uploading or transferring data to the cloud entails significant compliance risks and potential security breaches. Local deployment is often the only viable solution.

• Real-Time Performance: Scenarios like industrial control and on-site operations demand millisecond-level response times. Cloud APIs, susceptible to network fluctuations and unpredictable latency, fail to meet stringent real-time business requirements.

• Long-Term Cost: The token-based billing model for frequently accessing cloud-based large models results in cumulative costs that far exceed the one-time investment in local hardware. The larger the business scale, the more pronounced this cost disparity becomes.

The watershed for model capability lies precisely at 20 billion parameters and above. While 7B models are sufficient for simple Q&A or text summarization, handling complex tasks like logical reasoning, code generation, multi-step tool invocation, and long-context RAG requires the robust capabilities of 27B or 31B-scale models for genuine productivity. In the past, running such models was confined to expensive server-grade GPUs, making edge deployment virtually impossible.

Now, the multi-card cascade solution based on the RK1828 computing card has turned this “impossible” into a “feasible reality.”

4-Card Cascading: Redesigning the Edge Computing Architecture

Leveraging Rockchip’s latest official RK182X_AI_SDK_V1.1.0 technology stack, the RK1828 computing card now fully supports 4-card PCIe cascade inference, delivering a complete hardware-to-software deployment solution.

Hardware Foundation: Main Controller + 4 Cards, Linearly Scalable Computing Power

This solution utilizes the high-performance OK3588-C development board as the main controller. It connects four RK1828 AI accelerator cards via a high-speed PCIe bus, creating a compact edge AI inference cluster. This design fully leverages the scheduling capabilities of the OK3588-C board while seamlessly integrating the computational power of the four accelerators. The scaling method is straightforward, offering linear performance gains.

Software Coordination: Pipeline Parallelism for Efficient Multi-Card Task Division

Building on the comprehensive AI deployment software stack provided by the RKNN3 SDK, we have successfully implemented Pipeline Parallelism. This technique precisely segments the LLM model at Transformer layer boundaries into multiple segments, each independently deployed on a single RK1828 accelerator. The cards work in relay to complete the full inference pipeline, achieving peak collaborative efficiency.

Currently Supported Adaptations:

• Full 4-Card Configuration: Qwen3.5-27B, Qwen3.8-27B, Gemma-4-31B-it

• Lightweight 2-Card Configuration: Qwen3.5-9B, Gemma-4-12B-it

The system allows for flexible selection and configuration based on specific computational requirements.

Optimal Energy Efficiency: Low Power, High Precision

Energy efficiency is a critical metric for edge scenarios, and this solution delivers performance that far exceeds industry expectations.

Power Management: Each RK1828 card consumes only about 10W. Even under full load with four cards, the total power remains around 40W, eliminating the need for bulky cooling systems and making it ideal for edge hardware environments.

• Quantization Strategy: A refined hybrid quantization approach is employed: W4A16/group32 for Transformer layers, W6A16/group32 for the lm_head layer, while the final norm retains FP16 precision. This strategy dramatically reduces memory footprint while preserving model output accuracy to the greatest extent possible.

• Extensibility: The SDK natively supports LoRA fine-tuning and SpeedUP inference acceleration, providing developers with ample room for customization and optimization.

Practical Validation: Engineering Performance Testing of a 27B Model on the Edge

To evaluate the engineering performance of the OK3588-C with a 4-card RK1828 setup, we conducted an end-to-end test using the Qwen3.8-27B model. The test focused on two typical workloads: embedded development code generation and offline technical document retrieval.

Scenario 1: AI Programming Assistant for Enhanced Embedded Development Efficiency

Target Verification

Objective: To verify if a 27B model can handle complex engineering code generation and serve as an efficiency tool for embedded R&D.

Task: Using the rknn-toolkit2 on an RK3588 platform to deploy a YOLOv8n model, write a complete Python-based asynchronous inference service.

Results: The model generated a complete code framework in one go, covering model loading, image pre-processing, asynchronous task queues, inference scheduling, result post-processing, service interfaces, and exception handling.Hundreds of lines of code were output in a stream, with coherent context, clear module division, and rigorous call logic throughout. First-token latency was only 819 ms, with a generation speed of up to 13 tokens/s, delivering a smooth interactive experience.

Scenario Value:

The on-device 27B model is no longer just a “chat tool” but an AI programming assistant that can be integrated into the R&D workflow. It significantly shortens the development cycle from requirements to prototypes in tasks like model deployment, API encapsulation, sample development, and troubleshooting—making it particularly suitable for resource-constrained edge development scenarios.

Scenario 2: Product Manual RAG for Intelligent Technical Support

Objective:

To verify whether the large model, when combined with local document RAG (Retrieval-Augmented Generation), can be quickly transformed into a dedicated technical support tool that provides traceable and verifiable answers. Task: Using the “RK182X_AI SDK User Manual” as a local knowledge base to implement an intelligent Q&A system for product documentation.

Example Questions:

Introduce the development framework of the RK1820/RK1828 platform.
What is the reset procedure for the RK1820/RK1828?

Test Results

The retrieval module accurately located the corresponding manual sections, fully returning operational steps, command lines, and corresponding PDF page numbers.The 27B model synthesized the retrieved information to output well-structured and clearly organized answers. The interface simultaneously displayed the RAG source hits, making all answers traceable and verifiable. First-character response time was approximately 1.2 seconds, with a generation speed of about 13 tokens/s, achieving a level suitable for daily interactive use.

Scenario Value:

No model retraining required. By adopting the “local document retrieval + large model generation” paradigm, SDK documents, development guides, and maintenance manuals can be quickly transformed into interactive intelligent Q&A systems. All data remains entirely local throughout the process, improving technical support efficiency while ensuring document data security.

4. Summary and Outlook

From validating general capabilities to deploying them in industry-specific scenarios, the 4-card RK1828 cascade solution has successfully enabled the complete local execution of 27B/31B large models at the edge, delivering four core values for embedded AI:

• Computing Power Breakthrough

  Run 27B/31B large models locally at the edge, unleashing complex reasoning capabilities and completely eliminating reliance on cloud computing power.

• Extreme Energy Efficiency

  Support for hundred-billion-parameter models with approximately 40W power consumption offers a highly cost-effective advantage for private deployments.

• Flexible Scalability

  Supports pipeline parallelism, hybrid quantization, and LoRA fine-tuning to adapt to customized requirements across different industries.

• Scenario Closure

  It covers three typical scenarios: AI programming, product Q&A, and industry knowledge bases, and its implementation ability has been verified.

This solution achieves stable inference of 27B/31B large - language models at the edge through the PCIe cascade cooperation between OK3588 - C and four RK1828 units. With a co - processor power consumption budget of approximately 40W, it provides a cost - effective hardware reference design for local deployments with strict data security requirements and a pursuit of deterministic latency.

Evaluation Support:

• Applied for the OK3588 - C and RK1828 cascade evaluation kit and hardware specification.

• Acquired the RKNN3 distributed deployment toolchain, model quantization scripts, and reference examples.

• Contact the engineering and technical team to get deployment evaluation support for edge large models.

  

---