Run your own models.
AI as a swappable layer: open weights on your own hardware, no API key required.
Stop renting: Hosted-only model APIs
The ranking rule
Repositories matching this query, sorted by stars:
topic:llm-inference stars:>200 pushed:>2025-10-29 archived:falseSynced nightly; star history kept so momentum is measured, not guessed.
30 projects
live from source · api.github.com · Oct 29, 12:56 AM UTCray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
gitleaks
Find secrets with Gitleaks 🔑
llm-action
本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
litgpt
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
OpenLLM
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
openvino
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
PowerInfer
High-speed Large Language Model Serving for Local Deployment
kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
dynamo
A Datacenter Scale Distributed Inference Serving Framework
lmdeploy
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
openevolve
Open-source implementation of AlphaEvolve
ai-infra-book
《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
plano
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
turbo-fieldfare
Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook
flashinfer
FlashInfer: Kernel Library for LLM Serving
kserve
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
cactus
Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.
lemonade
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
shimmy
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Awesome-LLM-Inference
📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉
eko
Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai
RuVector
RuVector provides High Performance, Real-Time decisions and agent memory , Self-Learning Ai, Vector GNN DB built in Rust.
optillm
Optimizing inference proxy for LLMs
GenerativeAIExamples
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
Rapid-MLX
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
AI-Engineer-Headquarters
A collection of scientific methods, processes, algorithms, and systems to build stories & models.
ai-engineering-interview-questions
Your Cheat Sheet for AI Engineering Interview – Questions and Answers.
distributed-llama
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
No stories tagged to this direction recently.