LattePanda Mu Ultra 256V Local LLM Inference Evaluation

Test Scope: Results are based on internal testing under the specified hardware and software configuration. Actual performance may vary depending on model version, quantization method, drivers, runtime settings, cooling, and deployment environment. 

Overview

As large language models move toward on-device deployment, developers need compact computing platforms capable of processing text and visual inputs locally while keeping data under their control. This evaluation explores the local AI inference performance of the LattePanda Mu Ultra compute module, powered by the Intel Core Ultra 7 256V processor.

Nine model configurations from the Qwen3.5, Qwen3, Phi, and Gemma 4 families were evaluated using OpenVINO, INT4 quantization, and Intel integrated graphics. The tests examine time to first token (TTFT), prompt-processing throughput, text-generation throughput, long-context performance, visual-processing throughput, platform power consumption, and operating temperature.

Key Result: Qwen3-4B achieved the highest short-prompt prefill throughput and the lowest TTFT, while Qwen3.5-2B delivered the highest decode throughput. Overall, LattePanda Mu Ultra successfully ran all tested 2B- to 9B-class quantized language and multimodal models locally.

Test Platform and Method

Software Environment

Each model was evaluated using an English prompt of approximately 750 tokens discussing the differences between classical and quantum computing. The tests measured time to first token (TTFT), prefill and decode throughput, multimodal visual-processing performance, peak platform power, and peak temperature.

• All models were tested under the same core hardware and software configuration.
• Platform power represents the complete test platform, including the cooling fan, compute module, carrier board, and onboard components.
• Idle platform power was approximately 7 W.
• Results reflect inference performance rather than model accuracy, reasoning capability, or output quality.

Short-Prompt Text Performance

Decode Throughput

Qwen3.5-2B achieved the highest decode throughput at 55.45 tokens/s. Five configurations reached approximately 29 tokens/s or higher, while the tested 8B- and 9B-class models completed generation at approximately 18-19 tokens/s. Model architecture, quantization, runtime implementation, and memory behavior all influence throughput; parameter count alone does not determine speed.

Time to First Token

All recorded short-prompt TTFT values were below 0.81 seconds. Qwen3-4B responded fastest at 0.223 seconds. TTFT varies with input length, cache state, model loading, sampling configuration, and system load.

Prompt Processing

Qwen3-4B achieved the highest prefill throughput at 3,381.68 tokens/s, enabling faster processing of long prompts, documents, and code. High prefill performance can help reduce waiting time in local knowledge-base, document-analysis, and RAG workflows.

Long-Context Performance

Five representative configurations were tested with actual 8K- and 16K-token inputs. Longer inputs had a much larger effect on TTFT than on decode throughput in most of the recorded runs.

Qwen3.5-2B maintained the highest decode throughput in the long-context test and recorded a TTFT of 5.17 seconds with a 16K-token input. Qwen3.5-4B showed almost no change in decode throughput as the input increased from 8K to 16K tokens, although its TTFT increased from 8.24 to 16.52 seconds.

Qwen3-4B recorded a smaller relative TTFT increase, from 22.04 to 23.15 seconds. However, its TTFT was already relatively high at 8K, so the smaller percentage change does not necessarily indicate better long-context performance.

Test Note: These results reflect performance with the tested 8K- and 16K-token inputs. Performance at each model’s maximum supported context length may differ.

Multimodal Processing

Five multimodal model configurations completed the visual-processing test. Qwen3.5-2B achieved the highest throughput at 11.16 FPS, followed by Qwen3.5-4B at 8.63 FPS and Qwen3.5-9B at 7.22 FPS. Under the tested configuration, all three Qwen3.5 models outperformed the two Gemma 4 models in visual-processing throughput.

Test Note: Vision FPS represents the visual-processing throughput recorded by the benchmark tool. It may not include the complete application pipeline, such as image preprocessing, language generation, data transfer, and post-processing. End-to-end performance may vary depending on the application configuration.

Platform Power and Thermal Performance

Platform Power

Peak platform power remained between approximately 26 W and 30 W across all tested model configurations, while idle power was approximately 7 W. These results demonstrate local INT4 language and multimodal model inference within a compact platform power envelope.

Power measurements represent the complete test platform.

Performance-Based Model Guidance

The following guidance is based on measured speed and platform readings only. Final selection should also evaluate task accuracy, reasoning quality, language performance, memory use, and stability.

A tendency for Qwen3.5-4B to continue generating internal reasoning without promptly returning a final response was observed in one setup. This should be treated as configuration-specific until reproduced with the exact model revision, prompt template, sampling settings, and output limit.

Built for Practical On-Device AI

Powered by the Intel Core Ultra 7 256V, LattePanda Mu Ultra successfully ran all tested 2B- to 9B-class INT4 language and multimodal models locally using OpenVINO and Intel integrated graphics. Qwen3-4B delivered the fastest prompt processing and initial response, while Qwen3.5-2B led both text-generation speed and visual-processing throughput.

The tests also demonstrate Mu Ultra’s ability to handle 8K- and 16K-token inputs within a compact platform power envelope. With peak power at approximately 30 W or below and recorded temperatures below 65°C under active cooling, Mu Ultra brings practical local AI capabilities to a compact x86 compute module.

From private AI assistants and document-processing workflows to multimodal applications and edge-AI development, Mu Ultra gives developers a flexible platform for building and deploying AI directly on the device.

Test Disclaimer

All benchmark data was collected through internal testing under the specified hardware and software configuration. Actual performance may vary depending on model version, quantization method, drivers, runtime settings, cooling solution, and deployment environment.

References

• Intel Core Ultra 7 256V Specifications
• Qwen3.5 Official Model Collection
• Google Gemma Official Model Collection
• OpenVINO Generative AI Documentation