AgenticSwarmBenchpip install agentic-swarm-benchReal-world agentic inference benchmarks. Recorded coding sessions replayed against provider endpoints - measuring what matters as context grows beyond 100K tokens.
All benchmarks use asb replay mode - a recorded agentic coding session replayed against each provider endpoint. Cost is derived from per-provider pricing applied to actual token usage.
Total session cost against decode speed or wall clock time
Total session cost against decode speed or wall clock time
Decode speed measured in output tokens per second per user
Decode speed ranked by median (p50) — higher is better
DeepSeek v4-flash
DeepSeek
GLM 5
Z.AI
GLM 5.1
Z.AI
Kimi K2.5
Moonshot
Kimi K2.6
Moonshot
GLM 4.6
Z.AI
DeepSeek v4-pro
DeepSeek
Output tokens/sec per user
How decode speed changes as context length grows
Decode speed as context grows
Time to First Token (TTFT) across context sizes
Best-case time to first token at each context size
Expand each provider to see their models and full benchmark breakdowns