AIPerf: Ultimate AI Agent Skill Benchmark Tool for Claude & Codex

AIPerf: AI Agent Skill Benchmark Tool for Claude & Codex

AIPerf is a vendor-neutral benchmarking tool designed for assessing AI agent skills' inference performance. It enables detailed evaluation of latency, throughput, and goodput across various endpoint types for AI coding agents like Claude Code, Cursor, and Codex.

What this skill does

• Replays production traces with precise timestamp accuracy using Mooncake, Bailian, and BurstGPT datasets.
• Measures goodput, emphasizing the percentage of requests meeting all service level objectives.
• Profiles performance with concurrent request handling, request rate patterns, and fixed-schedule trace replays.
• Offers user-centric and multi-run confidence analysis for robust performance insights.
• Supports 17 distinct endpoint types including chat, completions, embeddings, and image generation.
• Utilizes 10 custom dataset formats and over 20 public datasets.
• Integrates with GPU telemetry and Prometheus for detailed performance monitoring.
• Includes extensibility features via plugins, allowing custom endpoints, datasets, and metrics.

Who it is for

This tool is intended for developers and operators who require rigorous benchmarking of AI coding agents. Teams working with Claude Code, Cursor, and Codex can benefit from its precise performance evaluations.

Use cases

• Developers measuring AI agent skill performance against OpenAI-compatible inference servers such as vLLM, SGLang, and TensorRT-LLM.
• Operators seeking defensible and detailed latency, throughput, and goodput metrics.
• Teams needing to benchmark AI agent capabilities using production-like workloads.

Technical details

• AIPerf is a successor to NVIDIA's genai-perf, crafted by the AI-Dynamo team, ensuring a vendor-neutral benchmarking approach.
• The tool supports various endpoint types, including chat, completions, embeddings, and image and video generation.
• Customizable datasets formats include single_turn, multi_turn, mooncake_trace, and others.
• Integrates seamlessly with systems using GPUs and Prometheus for enhanced telemetry data collection.
• Accommodates custom extensions with plugins, offering adaptability for diverse benchmarking needs.

Source & Licence

This package is built on open-source work published by air-gapped (air-gapped/skills) and distributed under MIT. The original licence text and copyright notice are included in your download.

Personal and commercial use, modification and redistribution are permitted, provided the original copyright and licence notice are retained.

Your purchase covers curation, licence verification, packaging, documentation and instant delivery. It does not grant exclusive rights to the underlying open-source code, which remains available under its original licence.

Delivery & Support

• Delivery: instant — a secure download link is emailed to you as soon as payment is confirmed.
• Format: ZIP archive containing the skill files, documentation and the original licence.
• Support: support@mcpcart.com — we aim to reply within 2 business days.
• Updates: updates are included only where stated on this page.

Refunds

This is a digital product delivered immediately after purchase. By completing your order you request immediate delivery and acknowledge that, once the download has been accessed, the statutory right to cancel no longer applies to the extent permitted by law. Refund requests are handled in accordance with our published Refund Policy.

Claude, Codex, Gemini and Cursor are trademarks of their respective owners. MCP Cart is an independent marketplace and is not affiliated with, endorsed by, or sponsored by any of them. Compatibility references describe interoperability only.