{"product_id":"aiperf-ultimate-ai-agent-skill-benchmark-tool-for-claude-codex","title":"AIPerf: Ultimate AI Agent Skill Benchmark Tool for Claude \u0026 Codex","description":"\u003ch3\u003eAIPerf: AI Agent Skill Benchmark Tool for Claude \u0026amp; Codex\u003c\/h3\u003e\n\n\u003cp\u003eAIPerf is a vendor-neutral benchmarking tool designed for assessing AI agent skills' inference performance. It enables detailed evaluation of latency, throughput, and goodput across various endpoint types for AI coding agents like Claude Code, Cursor, and Codex.\u003c\/p\u003e\n\n\u003ch3\u003eWhat this skill does\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003eReplays production traces with precise timestamp accuracy using Mooncake, Bailian, and BurstGPT datasets.\u003c\/li\u003e\n  \u003cli\u003eMeasures goodput, emphasizing the percentage of requests meeting all service level objectives.\u003c\/li\u003e\n  \u003cli\u003eProfiles performance with concurrent request handling, request rate patterns, and fixed-schedule trace replays.\u003c\/li\u003e\n  \u003cli\u003eOffers user-centric and multi-run confidence analysis for robust performance insights.\u003c\/li\u003e\n  \u003cli\u003eSupports 17 distinct endpoint types including chat, completions, embeddings, and image generation.\u003c\/li\u003e\n  \u003cli\u003eUtilizes 10 custom dataset formats and over 20 public datasets.\u003c\/li\u003e\n  \u003cli\u003eIntegrates with GPU telemetry and Prometheus for detailed performance monitoring.\u003c\/li\u003e\n  \u003cli\u003eIncludes extensibility features via plugins, allowing custom endpoints, datasets, and metrics.\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003ch3\u003eWho it is for\u003c\/h3\u003e\n\u003cp\u003eThis tool is intended for developers and operators who require rigorous benchmarking of AI coding agents. Teams working with Claude Code, Cursor, and Codex can benefit from its precise performance evaluations.\u003c\/p\u003e\n\n\u003ch3\u003eUse cases\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003eDevelopers measuring AI agent skill performance against OpenAI-compatible inference servers such as vLLM, SGLang, and TensorRT-LLM.\u003c\/li\u003e\n  \u003cli\u003eOperators seeking defensible and detailed latency, throughput, and goodput metrics.\u003c\/li\u003e\n  \u003cli\u003eTeams needing to benchmark AI agent capabilities using production-like workloads.\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003ch3\u003eTechnical details\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003eAIPerf is a successor to NVIDIA's genai-perf, crafted by the AI-Dynamo team, ensuring a vendor-neutral benchmarking approach.\u003c\/li\u003e\n  \u003cli\u003eThe tool supports various endpoint types, including chat, completions, embeddings, and image and video generation.\u003c\/li\u003e\n  \u003cli\u003eCustomizable datasets formats include single_turn, multi_turn, mooncake_trace, and others.\u003c\/li\u003e\n  \u003cli\u003eIntegrates seamlessly with systems using GPUs and Prometheus for enhanced telemetry data collection.\u003c\/li\u003e\n  \u003cli\u003eAccommodates custom extensions with plugins, offering adaptability for diverse benchmarking needs.\u003c\/li\u003e\n\u003c\/ul\u003e\n\u003c!-- mcpcart:static-blocks:start --\u003e\n\u003chr\u003e\n\u003ch3\u003eSource \u0026amp; Licence\u003c\/h3\u003e\n\u003cp\u003eThis package is built on open-source work published by \u003cstrong\u003eair-gapped\u003c\/strong\u003e (\u003ca href=\"https:\/\/github.com\/air-gapped\/skills\" rel=\"nofollow noopener\" target=\"_blank\"\u003eair-gapped\/skills\u003c\/a\u003e) and distributed under \u003cstrong\u003eMIT\u003c\/strong\u003e. The original licence text and copyright notice are included in your download.\u003c\/p\u003e\n\u003cp\u003ePersonal and commercial use, modification and redistribution are permitted, provided the original copyright and licence notice are retained.\u003c\/p\u003e\n\u003cp\u003eYour purchase covers curation, licence verification, packaging, documentation and instant delivery. It does not grant exclusive rights to the underlying open-source code, which remains available under its original licence.\u003c\/p\u003e\n\u003ch3\u003eDelivery \u0026amp; Support\u003c\/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cstrong\u003eDelivery:\u003c\/strong\u003e instant — a secure download link is emailed to you as soon as payment is confirmed.\u003c\/li\u003e\n\u003cli\u003e\n\u003cstrong\u003eFormat:\u003c\/strong\u003e ZIP archive containing the skill files, documentation and the original licence.\u003c\/li\u003e\n\u003cli\u003e\n\u003cstrong\u003eSupport:\u003c\/strong\u003e \u003ca href=\"mailto:support@mcpcart.com\"\u003esupport@mcpcart.com\u003c\/a\u003e — we aim to reply within 2 business days.\u003c\/li\u003e\n\u003cli\u003e\n\u003cstrong\u003eUpdates:\u003c\/strong\u003e updates are included only where stated on this page.\u003c\/li\u003e\n\u003c\/ul\u003e\n\u003ch3\u003eRefunds\u003c\/h3\u003e\n\u003cp\u003eThis is a digital product delivered immediately after purchase. By completing your order you request immediate delivery and acknowledge that, once the download has been accessed, the statutory right to cancel no longer applies to the extent permitted by law. Refund requests are handled in accordance with our published Refund Policy.\u003c\/p\u003e\n\u003cp style=\"font-size:0.85em;color:#666;\"\u003eClaude, Codex, Gemini and Cursor are trademarks of their respective owners. MCP Cart is an independent marketplace and is not affiliated with, endorsed by, or sponsored by any of them. Compatibility references describe interoperability only.\u003c\/p\u003e\n\u003c!-- mcpcart:static-blocks:end --\u003e","brand":"MCP Cart","offers":[{"title":"Default Title","offer_id":52830569464119,"sku":"MCP-AIR-GAPPED-SKILLS-AIPERF","price":23.99,"currency_code":"GBP","in_stock":true}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/0981\/3950\/4951\/files\/F9h46F4TOXuFE7niu36m_f4d191ff856e4ba595a07d5683836e3e.jpg?v=1786619183","url":"https:\/\/mcpcart.com\/products\/aiperf-ultimate-ai-agent-skill-benchmark-tool-for-claude-codex","provider":"SPF PRO","version":"1.0","type":"link"}