In AI hardware product definition, the inference cost of edge small language models (SLMs) directly determines the PCBA BOM and user experience. The VocabX method recently shared by Hugging Face shows that pruning the vocabulary for specific scenarios can reduce inference cost by 77%. For IDH, this is not just algorithm optimization but a system engineering effort spanning chip selection to firmware architecture. This article provides an in-depth analysis of how to implement vocabulary specialization technology in real product forms such as LYAAI companion dolls, smart voice recorders, and AI badges, and presents a mass-production-ready device-cloud collaboration solution.
The parameter count of SLMs is typically concentrated in the embedding layer and the output layer (LM Head). Taking Qwen1.5-0.5B as an example, its vocabulary size is 151,936, and the embedding layer has approximately 0.5B×0.3=150M parameters. With INT8 quantization, it occupies 150MB, far exceeding the 8MB PSRAM of ESP32-S3. Through domain vocabulary pruning (e.g., retaining only 20,000 words), the embedding layer can be compressed to within 20MB, and further reduced to 10MB with INT4 quantization, making it easily affordable for RK3566 (1GB RAM), and even ESP32-S3 (8MB PSRAM) can run it in extremely narrow scenarios.
The output layer computation requires softmax over the entire vocabulary, and the vocabulary size directly determines the logits computation load. After pruning, the vocabulary is reduced from 150,000 to 20,000, reducing the output layer matrix multiplication by 87%. On RK3588, the measured first-token latency drops from 220ms to 45ms. For full-duplex voice interaction (target <800ms), this leaves ample budget for the acoustic front-end and network transmission.
ESP32-S3 features 8MB PSRAM and a 240MHz dual-core, suitable for wake-word + command-word scenarios. By specializing the vocabulary to 5,000 words, the model parameter count can be compressed to within 30M (INT8), with RAM usage of about 30MB—still exceeding on-chip PSRAM, requiring external PSRAM or Flash direct inference (XIP) optimization. It is recommended to use the ESP32-S3+8MB PSRAM version to run a vocabulary-specialized TinyLLM (e.g., a pruned TinyLlama-15M) for local intent understanding, with complex queries falling back to the cloud. In terms of power consumption, the inference current is about 120mA, and a dynamic frequency scaling strategy is needed to ensure the module surface temperature ≤38°C.
RK3566 (quad-core A55, 1GB RAM) can run a vocabulary-specialized 0.5B model (INT4), with RAM usage of about 300MB and power consumption of about 1.5W, requiring a graphene heat sink to control temperature rise. RK3588 (octa-core A76/A55, 4GB RAM) can host a 1.5B model, supporting more complex multi-turn dialogues, but attention must be paid to NPU utilization—after vocabulary specialization, the output layer computation decreases, making the NPU load more balanced.
JieLi JL7016 focuses on low-power audio processing with only a few hundred KB of RAM, and cannot directly run SLMs. However, it can serve as a voice front-end (noise reduction, VAD, wake-up), with the specialized vocabulary mapping table固化 in Flash to enable local command-word recognition, while complex tasks are handled by the main controller (e.g., RK3566) or the cloud. This heterogeneous architecture is particularly common in AI badges.
The target latency of 800ms needs to be decomposed into: acoustic front-end (50ms) + edge VAD/wake-up (30ms) + edge SLM first token (100ms) + network transmission (RTT 100ms) + cloud large model first token (200ms) + speech synthesis (150ms) + playback buffer (70ms). Vocabulary specialization reduces the edge first token from 220ms to 45ms, leaving margin for network fluctuations.
When the confidence of the edge SLM output is below a threshold, a cloud full-vocabulary large model is triggered for fallback. The firmware needs to implement synchronous updates of the vocabulary mapping table (edge vocabulary ID ↔ cloud vocabulary ID), supporting OTA differential upgrades. It is recommended to use a double-buffered mapping table to ensure seamless switching during fallback.
WebSocket full-duplex streaming transmission is adopted, with the edge continuously uploading audio features and the cloud streaming back text and audio. The edge SLM can process local intents in parallel, enabling "edge-cloud collaboration." On RK3588, the measured end-to-end latency is 650ms (4G network), meeting interaction requirements.
The module uses RK3566 + vocabulary-specialized 0.5B model (INT4), with RAM usage of 280MB, supporting offline dialogue and emotional interaction. The thermal design uses thermal silicone + metal casing to ensure surface temperature ≤38°C. The vocabulary focuses on domains such as children's stories and emotional Q&A, pruned to 30,000 words.
The main controller uses ESP32-S3 + external PSRAM, running a micro model with vocabulary specialized to 10,000 words to implement basic voice assistant functions. Display and voice are processed separately to reduce power consumption. The vocabulary mapping table is stored in Flash and supports OTA updates.
Adopts JieLi JL7016 + cloud collaboration architecture, with local processing of wake-up and command words, and the cloud responsible for semantic understanding. Vocabulary specialization is performed in the cloud, and the edge only needs to maintain the mapping table, reducing hardware cost.
The embedding and output layers are sensitive to quantization; it is recommended to use AWQ (Activation-aware Weight Quantization) to maintain accuracy, while INT8 can be used for intermediate layers. After vocabulary specialization, the embedding layer parameter count is reduced, lowering the calibration cost of AWQ. Measured INT4 AWQ on the specialized vocabulary shows <1% accuracy loss.
The mapping table needs to support dynamic loading and version management. A two-level mapping is recommended: local vocabulary ID → unified semantic ID → cloud vocabulary ID. The mapping table is stored in binary format, loaded into RAM at startup, with O(1) query complexity.
Recommended PCBA block diagram: main controller (RK3566/RK3588) + audio Codec (ES7210) + PSRAM (optional) + WiFi/BT module (RTL8723DS) + power management (AXP209). The vocabulary-specialized model is deployed on the main controller's NPU/CPU, and the cloud service is connected via MQTT/WebSocket.
Vocabulary specialization is a key means to reduce cost and improve efficiency for edge SLMs, but it must be deeply coupled with chip selection, thermal design, and device-cloud collaboration. IDH should establish a complete toolchain from corpus collection, vocabulary pruning, quantization deployment to firmware mapping, in order to build technical barriers in the red ocean of AI hardware.
ChipTalk Engineering
Deep Tech & Custom Silicon