From AnyLanguageModel to On-Device Agent Harness: Engineering Practice of RK3588 + ESP32-S3 Heterogeneous Collaboration

Introduction: Hardware Engineering Challenges of On-Device Agents

AnyLanguageModel 1.0 demonstrated the feasibility of local model runtime on Apple platforms, but its core idea—decoupling model inference from hardware resources and managing memory, power, and task scheduling through a lightweight Harness—is equally applicable to embedded heterogeneous platforms. For IDH, the real challenge lies in: how to smoothly deploy LLMs and voice interaction into AI companion dolls, waveguide glasses, or smart badges under limited RAM/Flash, stringent power budgets, and micro-chassis heat dissipation (≤38°C) constraints.

This article takes the heterogeneous architecture of Rockchip RK3588 and Espressif ESP32-S3 as an example to analyze the design points of the end-cloud streaming collaborative pipeline and provide a reusable Agent Harness reference implementation.

Heterogeneous Architecture Selection: Division of Labor Philosophy of RK3588 + ESP32-S3

Chip Specifications and Role Positioning

| Chip Model | Core Resources | Role | Key Constraints | |---------|---------|------|---------| | RK3588 | 4×A76 + 4×A55, 6TOPS NPU, 8GB LPDDR4X | Local LLM inference, speech recognition, intent understanding | Power 3-5W, requires active cooling or large heat spreader | | ESP32-S3 | Dual-core Xtensa LX7 @240MHz, 512KB SRAM, 8MB PSRAM | Low-power audio front-end, sensor hub, wake word detection | Power <100mW, no cooling pressure | | nRF5340 | Dual-core Cortex-M33, 1MB Flash, 512KB RAM | BLE/Thread connectivity, ultra-low power standby | Suitable for space-constrained scenarios like glasses temples | | JieLi JL7016 | Dedicated audio DSP, low-power Bluetooth | Audio codec, noise reduction | Cost-sensitive doll chassis |

Engineering Recommendation: For AI companion dolls, the RK3588 + ESP32-S3 combination is recommended. RK3588 handles heavy-load inference, while ESP32-S3 processes audio capture and preprocessing. The two collaborate via UART/SPI or Wi-Fi 6. If the product form is waveguide glasses, RK3566 (low-power version) paired with nRF5340 can be chosen to reduce overall power consumption and size.

Memory and Storage Budget

• RK3588: Running an INT4-quantized 1B-3B parameter model (e.g., Qwen1.5-1.8B-Chat) requires approximately 1.5-2.5GB of memory. It is recommended to reserve 4GB for the system and model, with the remainder for audio buffering and network stack. Model files are stored on eMMC (≥32GB) and loaded into NPU dedicated memory at startup.
• ESP32-S3: The wake word model (e.g., WakeNet) occupies <200KB RAM. An audio ring buffer of 32KB is recommended (16kHz, 16bit, mono, about 1 second). PSRAM is used to store intermediate results of speech feature extraction.

Quantization Scheme and Inference Optimization

INT4/INT8 Mixed Quantization Strategy

• LLM Main Body: Adopt AWQ (Activation-aware Weight Quantization) INT4 quantization to achieve 15-20 tokens/s generation speed on RK3588 NPU (1.8B model). Accuracy loss is controlled within 2% (compared to FP16).
• Speech Recognition Model: Use INT8-quantized streaming Conformer or Whisper-tiny to achieve real-time factor (RTF) <0.3 on NPU.
• Wake Word Model: Run INT8-quantized DS-CNN on ESP32-S3, with false wake-up rate <1 time/24h.

Quantization Toolchain: Rockchip provides RKNN-Toolkit2 supporting ONNX/PyTorch model conversion and quantization; ESP32-S3 can use the ESP-DL framework for 8-bit quantization.

Memory Reuse and Dynamic Loading

To reduce peak memory, staged loading can be adopted: during the wake-up phase, only the wake word model and audio front-end are loaded; during the recognition phase, the ASR model is loaded; during the generation phase, the ASR model is unloaded and the LLM is loaded. The RK3588 NPU driver supports time-division multiplexing of multiple models, requiring model lifecycle management in the Harness.

End-Cloud Streaming Collaborative Pipeline: Full-Duplex <800ms Latency Design

Pipeline Architecture

[Microphone Array] → ESP32-S3 (AEC/NS/VAD) → Wake Word Detection → Audio Stream (Opus 16kbps) → Wi-Fi/BLE → RK3588
                                                                         ↓
[Speaker] ← ESP32-S3 (Decode/Playback) ← Audio Stream ← RK3588 (TTS) ← LLM Generation ← ASR Result

Critical Path Latency Breakdown (Target <800ms):

| Stage | Latency Budget | Implementation Points | |------|---------|---------| | Audio Capture & Preprocessing | 20ms | ESP32-S3 hardware I2S + software AEC | | Network Transmission (Wi-Fi) | 30ms | Local Wi-Fi 6, UDP transmission | | ASR Recognition | 150ms | Streaming recognition, first-word latency <100ms | | LLM First Token Generation | 200ms | INT4 quantization + NPU acceleration | | TTS Synthesis | 100ms | Streaming TTS, first-packet latency <50ms | | Network Return & Playback | 50ms | Audio buffer 20ms | | Total | ~550ms | Margin reserved |

Engineering Recommendation: Adopt a full-duplex streaming protocol, i.e., simultaneously upload audio and receive TTS stream while the user speaks, enabling a "listen while speaking" interaction experience. ESP32-S3 must support full-duplex I2S and Wi-Fi concurrency. It is recommended to use ESP-IDF's esp_http_client and esp_websocket_client components.

End-Cloud Task Offloading Strategy

• Local Priority: Wake word, VAD, and simple intents (e.g., "turn on the light") are processed locally on ESP32-S3 or RK3588 with response <100ms.
• Cloud Fallback: Complex Q&A and multi-turn dialogues can be routed to cloud LLMs, but seamless switching must be ensured when the network is available. The Harness should implement a confidence threshold: when the local LLM output confidence is below the threshold, automatically forward to the cloud.

Heat Dissipation and Power Budget: Path to Achieving ≤38°C Micro-Chassis

Thermal Design Points

• RK3588 Cooling: In enclosed spaces such as doll chassis, graphene heat spreaders + micro fans (e.g., 5V 3010 fan) or phase change materials are required. Measurements show that during continuous LLM inference, the RK3588 junction temperature can reach 85°C. Through 1mm graphite sheet and metal外壳 conduction, the surface temperature can be controlled below 38°C (ambient 25°C).
• ESP32-S3 Cooling: Low power consumption, no active cooling needed, but avoid layout adjacent to RK3588 heat sources.
• Dynamic Power Management: The Harness should implement temperature-aware scheduling: when a temperature sensor (e.g., TMP117) detects the enclosure temperature >36°C, reduce LLM inference frequency or switch to a smaller model (e.g., 0.5B) to ensure surface temperature does not exceed 38°C.

Power Budget Example (AI Doll)

| Component | Typical Power | Peak Power | |------|---------|---------| | RK3588 (Inference) | 3W | 5W | | ESP32-S3 (Audio) | 80mW | 120mW | | Microphone Array | 10mW | 10mW | | Speaker (1W) | 0.5W | 1W | | Total | ~3.6W | ~6.1W |

Battery Life: With a 5000mAh battery (3.7V), continuous interaction lasts about 5 hours; standby (only ESP32-S3 listening) power consumption <100mW, battery life up to 7 days.

Real Product Forms and Engineering Implementation Recommendations

1. AI Companion Doll (LYAAI Type)

• Hardware Configuration: RK3588 + ESP32-S3 + dual microphones + 1W speaker + 5000mAh battery.
• Harness Design: ESP32-S3 runs FreeRTOS, responsible for audio capture, wake word, and VAD; RK3588 runs Linux, hosting ASR, LLM, TTS, and cloud communication. The two exchange control commands and audio streams via UART or SPI.
• Cooling: RK3588 with 1mm graphite sheet, thermally conductive plastic enclosure, and ventilation holes if structure permits.

2. Waveguide Glasses

• Hardware Configuration: RK3566 (low power) + nRF5340 + bone conduction speaker + 400mAh battery.
• Harness Design: nRF5340 handles BLE connectivity and sensor hub; RK3566 runs lightweight LLM (0.5B) and ASR. Due to space constraints, time-division model loading and dynamic voltage and frequency scaling (DVFS) are required.
• Cooling: Graphene film attached inside the temples, utilizing the metal frame for heat dissipation, surface temperature can be controlled below 35°C.

3. AI Smart Badge

• Hardware Configuration: ESP32-S3 (main control) + cloud collaboration, or RK3566 local inference.
• Harness Design: If pure ESP32-S3 is used, only wake word and audio front-end run, with LLM entirely in the cloud; if local inference is needed, RK3566 can be selected with model limited to 0.5B INT4.
• Cooling: The badge is close to the human body, ensuring surface temperature ≤38°C. Plastic enclosure + graphite sheet is recommended, and continuous inference time should be limited.

Conclusion: Reusable Agent Harness Reference Design

The local runtime concept of AnyLanguageModel is equally applicable in the embedded field. The key is to dynamically schedule models and tasks based on hardware resources. The RK3588 + ESP32-S3 heterogeneous architecture proposed in this article achieves <800ms full-duplex interaction and ≤38°C surface temperature through quantization, streaming pipeline, and temperature-aware scheduling, and can be reused for product forms such as AI dolls, glasses, and badges. When implementing, IDH should focus on memory budget, thermal design, and end-cloud switching strategy, and perform quantization tuning based on specific chip models.