MiniCPM5-2B on AX8860 (llama.cpp + ggml-axera)

ๅœจ็ˆฑ่Šฏ AX8860 ็š„ NPU ไธŠ็”จ llama.cpp ่ฟ่กŒ MiniCPM5-2B ๆ–‡ๆœฌๆจกๅž‹๏ผšๆ€่€ƒๅผ€ๅ…ณใ€ๅทฅๅ…ท่ฐƒ็”จ๏ผŒOpenAI ๅ…ผๅฎนๆœๅŠกใ€‚ MiniCPM5-2B on the Axera AX8860 NPU with llama.cpp: text model with a thinking switch, tool calling and an OpenAI-compatible server.

ๆ€ง่ƒฝ / Performance

decode 59.5 tokens/s
4k prompt prefill 6219 tokens/s๏ผˆ3907 token 0.63 s๏ผ‰
ไธŠไธ‹ๆ–‡ / context ๆœ€้•ฟ / up to 131072

sh bench.sh ๅค็Žฐ / reproduce๏ผˆ็”Ÿๆˆ 50 token / 50 generated tokens๏ผ‰๏ผš

prompt -c prompt ่€—ๆ—ถ / time decode tokens/s
5 token 8192 0.04 s 59.5
46 token 8192 0.05 s 59.2
3907 token 8192 0.63 s 51.9
5 token 131072 0.04 s 59.2
3907 token 131072 0.63 s 52.1

ๅฟซ้€Ÿๅผ€ๅง‹ / Quick start

sh run.sh "The capital of France is" 64     # ๆ–‡ๆœฌ่กฅๅ…จ / completion
sh run.sh -c                                # ๅ‘ฝไปค่กŒๅฏน่ฏ / chat
sh run.sh -s 8080                           # OpenAI ๅ…ผๅฎนๆœๅŠก / server
sh bench.sh                                 # ๆ€ง่ƒฝๅค็Žฐ / benchmark
sh bench.sh ppl                             # ๅ›ฐๆƒ‘ๅบฆ / perplexity

้œ€่ฆ AX8860๏ผŒSDK V0.5.1 ๆˆ–ๆ›ดๆ–ฐ๏ผ›HF ไธ‹่ฝฝ็š„ๆ–‡ไปถๆฒกๆœ‰ๆ‰ง่กŒๆƒ้™๏ผŒ็”จ sh ่ฐƒ็”จๅณๅฏ / needs an AX8860 with SDK V0.5.1+; call the scripts with sh.

ๆœๅŠก่ฏทๆฑ‚ / Server request:

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "็”จไธ€ๅฅ่ฏไป‹็ป้•ฟๆฑŸ"}],
  "chat_template_kwargs": {"enable_thinking": false}
}'

ๆ–‡ไปถ / Files

ๆ–‡ไปถ / File ่ฏดๆ˜Ž / Description
MiniCPM5-2B-Q8_0.gguf ๆจกๅž‹ๆƒ้‡๏ผˆๅฎ˜ๆ–น Q8_0๏ผ‰/ weights (official Q8_0)๏ผŒ2.68 GB๏ผŒmd5 57e37e3fd7758a9e4f7c687cf5c0e257
bin/ llama.cpp ๆฟ็ซฏ็จ‹ๅบ๏ผˆ้™ๆ€้“พๆŽฅ๏ผ‰/ board programs (static)
kernels/ NPU kernel 2 ไธช๏ผšprefillใ€decode๏ผˆๅซ่พ“ๅ‡บๅคด๏ผ‰+ manifest.json / two kernels: prefill and decode (with the output head) + manifest.json
run.sh๏ผŒchat_template.jinja ่ฟ่กŒ่„šๆœฌไธŽๆœๅŠก็ซฏๅฏน่ฏๆจกๆฟ / run script and the server's chat template
bench.sh ๅŠ prompt ๆ€ง่ƒฝไธŽๅ›ฐๆƒ‘ๅบฆๅค็Žฐ / benchmark and perplexity

่ฏดๆ˜Ž / Notes

  • ๆƒ้‡ๅœจๅŠ ่ฝฝๆ—ถ็”ฑ GGUF ่ฝฌๆˆ MXFP8๏ผŒKV ไธบ BF16๏ผ›Q8_0 ไธŽ F16 GGUF ้ƒฝๅฏๅŠ ่ฝฝ / weights are converted from the GGUF to MXFP8 at load, with a BF16 KV cache; both Q8_0 and F16 GGUFs load.
  • ไธ€ๅฅ— kernel ่ฆ†็›–ๅˆฐ 131072 ไธŠไธ‹ๆ–‡๏ผŒKV ๆŒ‰ -c ๅˆ†้…๏ผ›็”จ็Žฏๅขƒๅ˜้‡ CTX ่ฐƒๆ•ด / one kernel set covers up to 131072 tokens and allocates the KV by -c; set CTX to change it.
  • ๆ€่€ƒๅผ€ๅ…ณ็”จ chat_template_kwargs.enable_thinking๏ผ›ๅทฅๅ…ท่ฐƒ็”จไฝฟ็”จๅŒ…ๅ†… chat_template.jinja / toggle thinking with chat_template_kwargs.enable_thinking; tool calling uses the bundled chat_template.jinja.
  • ๆฒกๆœ‰ MTP ๅคด๏ผŒไธๆ”ฏๆŒ MTP ๆŠ•ๆœบ่งฃ็  / no MTP head, so no MTP speculative decoding.
  • run.sh -c ้œ€่ฆ็œŸๅฎž็ปˆ็ซฏ / chat mode needs an interactive terminal.
  • ่ฟ่กŒๆ—ถ็‹ฌๅ  NPU 4 ๆ ธ๏ผ›ๆœๅŠก็ซฏ -np 1 / takes all 4 NPU cores; the server runs -np 1.

ๆญฃ็กฎๆ€ง / Correctness

  • ๆ–‡ๆœฌ่กฅๅ…จใ€ๆœๅŠก็ซฏๅฏน่ฏไธŽๆ€่€ƒๅผ€ๅ…ณๅœจๆฟไธŠๅทฅไฝœๆญฃๅธธ / completion, server chat and the thinking switch work on the board.
  • PPL๏ผˆsh bench.sh ppl๏ผŒ-c 512๏ผ‰/ perplexity:
่ฏญ่จ€ / language -ub 1 -ub 4
ไธญๆ–‡ / zh 16.70 16.72
่‹ฑๆ–‡ / en 13.33 13.44

ๆบ็  / Sources

ๆ›ดๆ–ฐ / Changelog

  • 2026-11-15๏ผš้ฆ–ๆฌกๅ‘ๅธƒ / first release.
Downloads last month
20
GGUF
Model size
3B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AXERA-TECH/MiniCPM5-2B-AX8860-llama.cpp

Quantized
(139)
this model

Collection including AXERA-TECH/MiniCPM5-2B-AX8860-llama.cpp