fastllm
fastllm is a high-performance, backend-independent inference library. Supports tensor-parallel dense models and hybrid MOE inference; GPUs with 10G+ can run full DeepSeek. A dual-socket 9004/9005 server with one GPU runs the original full-precision model at 20tps single-concurrency; INT4 reaches 30tps single and 60+ concurrent.
Translated from Chinese · show originalhide original
fastllm是后端无依赖的高性能大模型推理库。同时支持张量并行推理稠密模型和混合模式推理MOE模型，任意10G以上显卡即可推理满血DeepSeek。双路9004/9005服务器+单显卡部署DeepSeek满血满精度原版模型，单并发20tps；INT4量化模型单并发30tps，多并发可达60+。
- Stars
- 5,091
- Forks
- 501
- + today
- +1
- Created
- 3y
Ranking data as of October 5, 2026 (UTC).
Star History
Today, hour by hour
Overview
fastllm is a high-performance, backend-independent inference library. Supports tensor-parallel dense models and hybrid MOE inference; GPUs with 10G+ can run full DeepSeek. A dual-socket 9004/9005 server with one GPU runs the original full-precision model at 20tps single-concurrency; INT4 reaches 30tps single and 60+ concurrent. It ranks #4120 on GitTiger, gaining +1 star on October 5, 2026 (UTC).
The project is written in C++ and has 501 forks. It was created 3y ago.
git clone https://github.com/ztxz16/fastllm.git
cd fastllm
# see README for setup