LattePanda Mu Ultra 226V Local LLM Inference Evaluation

Test Scope: The results presented in this article are based on internal testing conducted under the specified hardware and software configurations. Performance may vary depending on model versions, system settings, drivers, cooling conditions, and deployment environments.

Overview

As large language models move toward on-device deployment, developers need compact computing platforms capable of processing text and visual inputs locally while keeping data under their control. This evaluation examines the local AI inference performance of the LattePanda Mu Ultra compute module powered by the Intel Core Ultra 5 226V processor.

Nine model configurations from the Qwen3.5, Qwen3, Phi, and Gemma 4 families were evaluated using OpenVINO, INT4 quantization, and Intel integrated graphics. The tests cover time to first token (TTFT), prompt-processing throughput, text-generation throughput, visual-processing performance, platform power consumption, and operating temperature.

Key Result: Qwen3-4B achieved the highest prefill throughput and the lowest TTFT, while Qwen3.5-2B delivered the highest decode throughput. Overall, the results demonstrate that LattePanda Mu Ultra can locally run quantized language and multimodal models ranging from 2B to 9B parameters.

Test Platform and Method

Hardware

Item	Configuration
Compute module	LattePanda Mu Ultra
Processor	Intel Core Ultra 5 226V
Memory	16 GB LPDDR5X; approximately 15 GB available to the test system
Acceleration	Intel integrated GPU
Cooling	Active cooling

Software and Test Settings

Item	Configuration
Operating system	Ubuntu 24.04 LTS, x86-64
Python	3.12.3
Inference framework	OpenVINO 2026.2
Quantization	INT4
Inference device	Intel integrated GPU
Model capability	Up to 256K context, depending on the model; not tested at 256K
Measured prompt length	Approximately 0.76K English tokens
Visual input	320 × 240 pixels

Each model was evaluated using an English prompt of approximately 760 tokens discussing the differences between classical and quantum computing. The recorded metrics include time to first token (TTFT), prefill throughput, decode throughput, total response time, visual-processing performance for multimodal models, peak platform power, and peak temperature.

Platform power represents the complete test platform, including the cooling fan, compute module, carrier board and other onboard components.
Idle platform power was approximately 7 W.
All results were obtained under the same hardware and software environment to provide a consistent performance reference.
Performance may vary depending on the model version, quantization method, runtime settings, prompt length, and deployment environment.
The results reflect inference performance and do not represent a ranking of model accuracy, reasoning ability, or output quality.

Performance Summary

Model	Scale	TTFT (s)	Prefill (tok/s)	Decode (tok/s)	*Vision FPS**	Peak power	Type
Qwen3.5-2B	2B	0.38	2006.55	55.13	10.16	28.14 W	Multimodal
Qwen3.5-4B	4B	0.57	1334.83	27.83	8.62	28.71 W	Multimodal
Qwen3.5-9B	9B	0.71	1070.22	18.13	5.82	29.43 W	Multimodal
Qwen3-4B	4B	0.25	3026.44	31.52	—	27.46 W	Text
Qwen3-8B	8B	0.37	2066.12	19.03	—	28.51 W	Text
Phi-3.5-mini	3.8B	0.30	2490.71	35.70	—	25.29 W	Text
Phi-4-mini	3.8B	0.31	2450.65	30.17	—	27.11 W	Text
Gemma 4 E2B*	E2B	0.76	997.19	36.04	1.93	27.49 W	Multimodal
Gemma 4 E4B*	E4B	1.39	550.06	21.70	1.54	29.02 W	Multimodal

Gemma 4 E2B and E4B refer to the models’ active parameter classes rather than their total parameter counts.

Vision FPS represents the visual-processing throughput recorded by the test program and should not be interpreted as the frame rate of a complete end-to-end video-analysis pipeline.

Detailed Analysis

Fast Prompt Processing

Qwen3-4B delivered the highest prefill throughput at 3,026.44 tokens/s, approximately 21.5% faster than Phi-3.5-mini at 2,490.71 tokens/s. Phi-4-mini followed closely at 2,450.65 tokens/s, while Qwen3-8B and Qwen3.5-2B both exceeded 2,000 tokens/s.

High prefill throughput enables faster processing of long prompts, documents, and code, helping reduce waiting time in applications such as local knowledge bases, document analysis, and RAG workflows.

Smooth Text Generation

Qwen3.5-2B achieved the highest decode throughput at 55.13 tokens/s. Gemma 4 E2B, Phi-3.5-mini, Qwen3-4B, and Phi-4-mini all reached approximately 30 tokens/s or higher, providing smooth text generation under the tested configuration.

The larger Qwen3-8B and Qwen3.5-9B models achieved 19.03 and 18.13 tokens/s respectively. In general, smaller models provide faster generation, while larger models require more computation. Actual performance also depends on model architecture, quantization, and runtime optimization.

Responsive Time to First Token

All nine configurations recorded a TTFT below 1.4 seconds using the approximately 760-token test prompt. Qwen3-4B responded fastest at 0.25 seconds, followed by Phi-3.5-mini at 0.30 seconds and Phi-4-mini at 0.31 seconds.

These results demonstrate responsive local interaction across all tested models. TTFT may vary depending on prompt length, model configuration, cache status, and system load.

Multimodal Processing

Model	Reported Vision FPS	Relative to Qwen3.5-2B
Qwen3.5-2B	10.16	100%
Qwen3.5-4B	8.62	84.8%
Qwen3.5-9B	5.82	57.3%
Gemma 4 E2B	1.93	19.0%
Gemma 4 E4B	1.54	19.0%

Five multimodal model configurations completed the visual-processing test. Qwen3.5-2B achieved the highest throughput at 10.16 FPS, followed by Qwen3.5-4B at 8.62 FPS and Qwen3.5-9B at 5.82 FPS. Under the tested configuration, all three Qwen3.5 models outperformed the two Gemma 4 models in visual-processing throughput.

Test Note: Vision FPS represents the visual-processing throughput recorded by the benchmark tool. It may not include the complete application pipeline, such as image preprocessing, language generation, data transfer, and post-processing. End-to-end performance will vary depending on the application configuration.

Power and Thermal Observations

Efficient Platform Power

Peak platform power ranged from 25.29 W to 29.43 W across the nine tested model configurations, with most results falling between 27 W and 29 W. Idle platform power was approximately 7 W.

These figures represent the complete test platform, including the cooling fan, compute module, carrier board, and other onboard components, rather than processor-only power consumption. The results demonstrate that Mu Ultra can run local INT4 language and multimodal models within a compact platform power envelope.

Stable Thermal Performance

With active cooling, the tested setup maintained stable temperatures throughout the benchmark runs under the tested workloads.

Test Note: Power and temperature results were recorded using the current active-cooling configuration. Actual performance may vary depending on ambient temperature, cooling design, power settings, enclosure, and workload duration.

Model Selection Guidance

The table below is based only on the recorded performance metrics. It is not a recommendation based on reasoning quality, accuracy, hallucination rate, language quality, or task success.

Priority	Candidate to evaluate	Recorded basis
Low TTFT and high prefill throughput	Qwen3-4B	0.25 s TTFT; 3026.44 tok/s prefill
Maximum text-generation throughput	Qwen3.5-2B	55.13 tok/s decode
Strong recorded text-generation speed	Phi-3.5-mini	35.70 tok/s decode; lowest recorded platform peak power
Highest recorded visual metric	Qwen3.5-2B	10.16 reported Vision FPS
Testing larger local models	Qwen3-8B / Qwen3.5-9B	8B- and 9B-class INT4 inference completed
Gemma ecosystem evaluation	Gemma 4 E2B	36.04 tok/s decode in the recorded setup

The reported tendency of Qwen3.5-4B to remain in a thinking loop should be documented as an observation tied to the exact model revision, prompt template, sampling settings, and maximum output length. It should not be presented as a universal property of the model until reproduced under controlled settings

Conclusion

Powered by the Intel Core Ultra 5 226V, LattePanda Mu Ultra successfully ran all tested 2B- to 9B-class INT4 language and multimodal models locally using OpenVINO and Intel integrated graphics.
Qwen3-4B delivered the highest prefill throughput and fastest time to first token, while Qwen3.5-2B achieved the highest text-generation speed and visual-processing throughput.
Even the larger 8B- and 9B-class models completed local inference at practical generation speeds, giving developers greater flexibility when balancing model size and performance.
Peak platform power remained below 30 W under the tested conditions.
With its compact x86 compute-module design and support for Ubuntu and OpenVINO, Mu Ultra provides a flexible platform for local AI assistants, document-processing workflows, multimodal applications, and edge-AI development.

Actual performance may vary depending on model version, software configuration, cooling, and deployment environment.

Test Disclaimer

All benchmark results were obtained through internal testing under the specified hardware and software configuration. Actual performance may vary depending on model version, quantization method, driver version, runtime settings, cooling solution, and deployment environment.