Short answer

Judge an edge platform by memory before TOPS. A 6 TOPS board like FET3588J-UP5 ( RK3588, up to 32 GB LPDDR4) handles multi-stream camera inference, but a useful language model needs dedicated high-bandwidth memory: the RK1820/RK1828 M.2 card pairs 20 TOPS with 5 GB of 3D stacked in-package DRAM for 3B–7B models, and four cascaded cards reach 27B–31B at roughly 40 W total.

Why keep inference on the device?

Forlinx cites three forces that push inference back to the field: physical data isolation for regulated industries such as industrial manufacturing, rail and finance; millisecond response for control loops where WAN jitter is unacceptable; and the cumulative cost of token billing under high concurrency, which eventually exceeds a one-time hardware purchase.

The catch is that "local" is not binary. A 3B model on an edge box and a 31B model beside a production line are different hardware problems, separated by a specification most selection tables do not put first.

Why memory, not TOPS, is the real ceiling

TOPS measures how fast a fixed-point array executes, assuming the data is already where it needs to be. A transformer decoder is dominated by weight streaming, so the binding constraint is whether the weights fit and how fast they can be fed. Forlinx states this on the RK1820/RK1828 product page: the card carries 5 GB of 3D stacked in-package DRAM specifically to remove external DDR bottlenecks and the edge AI "memory wall", and to stop the accelerator contending with the host for bandwidth.

That reframes the comparison. In Forlinx's catalogue the 6 TOPS tier covers six modules, but only the RK3588 family reaches 32 GB of LPDDR4; the RK3576 modules stop at 8 GB. The i.MX 9352 modules ship with 1 GB, and the RV1126B modules with 1–4 GB. Those numbers suit vision pipelines, where activations are small and reuse is high. They are not a foundation for a language model, however the TOPS figure reads.

What can each NPU class actually run?

Each tier below is tied to a workload Forlinx has documented in a product page or a published deployment article — the honest way to read a TOPS number.

NPU class	Documented workload	Representative platform
0.5 TOPS	Entry-level inference on NXP i.MX 9352	FET-MX9352-C
1 TOPS	YOLOv5 RKNN deployment and classification	OK3568-C (RK3568)
2 – 2.3 TOPS	YOLOv8n via an M.2 accelerator card; offline text-to-speech	FET-MX9596-C, FETMX8MPx-C
3 TOPS	Camera-first vision nodes	FET1126Bx-S (RV1126B)
6 TOPS	Real-time YOLOv8 object detection	FET3588J-UP5, FET3576-C
20 TOPS	3B–7B LLM and VLM inference	RK1820 / RK1828 M.2
4 × 20 TOPS	27B–31B LLM inference, ~40 W	OK3588-C + 4 × RK1828

Note the 2–2.3 TOPS row. The published YOLOv8n walkthrough for OK-MX9596-C targets the FAI-ARA240-M accelerator card, not the i.MX 9596 NPU alone. The on-board NPU runs the pipeline and pre-processing; the model that needs bandwidth moves onto a card with its own memory.

How much memory does a model need?

Model scale	Documented configuration	Memory provision	Interface
3B – 7B	RK1820 or RK1828, single card	5 GB 3D stacked in-package DRAM	M.2 2280 M-Key, PCIe
9B – 12B	RK1828 × 2, Qwen3.5-9B / Gemma-4-12B-it	2 × 5 GB in-package DRAM	PCIe
27B – 31B	RK1828 × 4, Qwen3.5-27B / Gemma-4-31B-it	4 × 5 GB, ~40 W cluster	PCIe

Two points matter. The 7B ceiling that limited on-device deployment is a memory-and-bandwidth problem, and it is the reason Forlinx gives for 27B and 31B models previously staying on server GPUs. Capacity alone is also not enough: the four-card setup splits the model at Transformer layer boundaries and runs the segments as a pipeline, so cards pass activations rather than duplicating weights. Scaling is linear because each card contributes a stage.

Which host board should carry the accelerator?

The host does not need the largest NPU, because the accelerator owns the model. It needs memory headroom, PCIe lanes and a thermal envelope that survives an industrial cabinet.

Module	SoC	CPU	NPU	Memory	Package	Best for
FET3588J-UP5	RK3588	—	6 TOPS	4–32 GB LPDDR4	LGA 624-pin, 50 × 50 mm	LLM clusters
FET3576-C	RK3576	4×A72 2.2 GHz + 4×A53 2.0 GHz	6 TOPS	2–4 GB LPDDR4	BtB, 4×100-pin	Vision only
FET-MX9596-C	i.MX 9596	6×A55 1.8 GHz + M7 + M33	2 TOPS	4–8 GB LPDDR4X	BtB, 4×80-pin	NXP programs
FET1126Bx-S	RV1126B	4×A53 1.6 GHz	3 TOPS	1–4 GB LPDDR4	Stamp-hole + LGA, 237 pins	Camera nodes
FET-MX9352-C	i.MX 9352	2×A55 1.7 GHz + M33	0.5 TOPS	1 GB LPDDR4	BtB, 2×100-pin	Entry-level

BtB is a board-to-board connector; a dash means the product page does not list the core configuration separately.

Only the RK3588 row has enough addressable memory to serve as an LLM host, and only it documents the PCIe cascade path.

When the on-board NPU is not enough

Two accelerator cards cover different halves of the problem and are not interchangeable.

Card	AI performance	Memory	Documented workload
RK1820 / RK1828	20 TOPS INT8, INT4/INT8/FP16 mixed precision	5 GB 3D stacked in-package DRAM	3B–7B LLM and VLM
FAI-ARA240-M	40 eTOPS, NXP Ara240 DNPU	8 GB or 16 GB LPDDR4	YOLOv8n deployment on OK-MX9596-C

Choose by workload shape. The RK1828 is built for transformer weight streaming and carries its own stacked DRAM, isolating the model from host memory pressure. The ARA240-M offers twice the headline AI throughput and more DRAM, suiting dense vision and multi-stream analytics. The RK1820/RK1828 lists RK3568, RK3576, RK3588 and RK3572 as supported hosts, so an existing vision board can often be upgraded in place.

What does the software side require?

The four-card configuration runs on Rockchip's RK182X_AI_SDK_V1.1.0 stack and the RKNN3 SDK, which implements the pipeline parallelism above. Quantization is mixed: W4A16 with group size 32 for the Transformer layers, W6A16 with group size 32 for the lm_head — a split that protects output-layer accuracy while keeping most weights at four bits.

Which configuration fits which project?

• Camera inspection, no language model: FET1126Bx-S for a single sensor node; FET3576-C when several streams share one 6 TOPS NPU.
• NXP-based program with light inference: FET-MX9596-C, adding an ARA240-M card only if a vision model outgrows 2 TOPS.
• 7B assistant on a machine: FET3588J-UP5 plus one RK1828 card. One slot, no cluster.
• 27B–31B reasoning or long-context RAG: OK3588-C with four RK1828 cards, planned around a 40 W accelerator budget and the airflow it implies.

FAQ

Can a 6 TOPS NPU run a 7B language model?

Not on its own. Forlinx documents 7B-scale inference as where edge deployments historically stalled, and the RK1828 is what lifts it, using 5 GB of in-package DRAM alongside its 20 TOPS NPU. A 6 TOPS SoC feeds that card; it does not run the model.

How much DRAM does on-device inference need?

Capacity must hold the quantized weights plus the KV cache for your context length. Use the documented pairings rather than extrapolating: 5 GB of dedicated stacked DRAM for 3B–7B, two cards for 9B–12B, four for 27B–31B. Host LPDDR4 is slower and contends with the CPU, which is why the accelerator carries its own memory.

Do I need PCIe to add an accelerator card?

Yes. Both the RK1820/RK1828 and the FAI-ARA240-M use M.2 over PCIe, and the RK1828 is specified for RK3568, RK3576, RK3588 and RK3572 hosts. Confirm lane count and card power before fixing the carrier board design.

Is on-device inference slower than a cloud API?

Per-token latency is often lower because there is no WAN round trip, which is the argument Forlinx makes for control-loop and field scenarios. Throughput per dollar still favours the cloud at low utilization, so the crossover is driven by concurrency and data sensitivity rather than raw speed.

Can one board run a 27B model?

Only in the documented four-card cascade, at roughly 40 W for the accelerator cluster. A single RK1828 tops out in the 3B–7B range. Treat 27B and 31B as a system design with airflow, PCIe topology and power budgeting.

Next step

If model size is settled, the rest is mechanical: confirm the host's PCIe lanes, size the thermal budget, and validate the quantization recipe against your accuracy targets. Start with the FET3588J-UP5 RK3588 module and the RK1820/RK1828 M.2 computing card. For camera-only designs, the FET1126Bx-S RV1126B module and FET-MX9596-C i.MX 9596 module cover the lower tiers, and the OK3588-C board is the documented platform for the four-card cluster.

---