{"product_id":"create-custom-graders-with-ai-skillevaluator-byog-byot","title":"Create Custom Graders with AI: SkillEvaluator BYOG\/BYOT","description":"\u003ch3\u003eCustom Grader Creation with AI: SkillEvaluator BYOG\/BYOT\u003c\/h3\u003e\n\n\u003cp\u003eSkillEvaluator BYOG\/BYOT is designed to assist developers in creating custom graders from existing benchmarks, rubrics, or verification tasks. This skill transforms pre-defined benchmark definitions into executable SkillEvaluator custom graders and native Harbor tasks, facilitating efficient AI agent evaluation.\u003c\/p\u003e\n\n\u003ch3\u003eWhat this skill does\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003eConverts team-owned benchmark definitions into runnable custom graders.\u003c\/li\u003e\n  \u003cli\u003eTransforms rubrics into Python or Shell evaluation scripts (`evals\/grader.py` or `evals\/grader.sh`).\u003c\/li\u003e\n  \u003cli\u003eEnables the addition of custom metrics alongside default evaluator metrics.\u003c\/li\u003e\n  \u003cli\u003eConverts task files such as `task.yaml`, `task.json`, pytest checks, or shell verifiers into BYOG\/BYOT formats.\u003c\/li\u003e\n  \u003cli\u003eValidates that a team can implement and run its benchmark through SkillEvaluator.\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003ch3\u003eWho it is for\u003c\/h3\u003e\n\u003cp\u003eThis skill is aimed at developers and teams using AI coding agents like Claude Code, Cursor, and Codex. It benefits those who need to incorporate custom evaluation logic within their AI agent development workflows.\u003c\/p\u003e\n\n\u003ch3\u003eUse cases\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003eBringing an existing benchmark into the SkillEvaluator environment for enhanced evaluation capabilities.\u003c\/li\u003e\n  \u003cli\u003eBuilding custom grading scripts to assess AI agent performance using non-standard metrics.\u003c\/li\u003e\n  \u003cli\u003eEnsuring that team benchmarks can be executed and validated through SkillEvaluator.\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003ch3\u003eTechnical details\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003eUtilizes agent-evaluation and aiapplication tools for effective grader customization.\u003c\/li\u003e\n  \u003cli\u003eSupports BYOG\/BYOT (Bring Your Own Grader\/Task) integration for flexible grader logic.\u003c\/li\u003e\n  \u003cli\u003eProvides modes such as `default_plus_custom` and `custom_only` to tailor evaluation logic to specific needs.\u003c\/li\u003e\n\u003c\/ul\u003e\n\u003c!-- mcpcart:static-blocks:start --\u003e\n\u003chr\u003e\n\u003ch3\u003eSource \u0026amp; Licence\u003c\/h3\u003e\n\u003cp\u003eThis package is built on open-source work published by \u003cstrong\u003eNVIDIA\u003c\/strong\u003e (\u003ca href=\"https:\/\/github.com\/NVIDIA\/SkillEvaluator\" rel=\"nofollow noopener\" target=\"_blank\"\u003eNVIDIA\/SkillEvaluator\u003c\/a\u003e) and distributed under \u003cstrong\u003eApache-2.0\u003c\/strong\u003e. The original licence text and copyright notice are included in your download.\u003c\/p\u003e\n\u003cp\u003ePersonal and commercial use, modification and redistribution are permitted under the Apache License 2.0, which also includes an express patent grant. Attribution and any NOTICE file must be retained.\u003c\/p\u003e\n\u003cp\u003eYour purchase covers curation, licence verification, packaging, documentation and instant delivery. It does not grant exclusive rights to the underlying open-source code, which remains available under its original licence.\u003c\/p\u003e\n\u003ch3\u003eDelivery \u0026amp; Support\u003c\/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cstrong\u003eDelivery:\u003c\/strong\u003e instant — a secure download link is emailed to you as soon as payment is confirmed.\u003c\/li\u003e\n\u003cli\u003e\n\u003cstrong\u003eFormat:\u003c\/strong\u003e ZIP archive containing the skill files, documentation and the original licence.\u003c\/li\u003e\n\u003cli\u003e\n\u003cstrong\u003eSupport:\u003c\/strong\u003e \u003ca href=\"mailto:support@mcpcart.com\"\u003esupport@mcpcart.com\u003c\/a\u003e — we aim to reply within 2 business days.\u003c\/li\u003e\n\u003cli\u003e\n\u003cstrong\u003eUpdates:\u003c\/strong\u003e updates are included only where stated on this page.\u003c\/li\u003e\n\u003c\/ul\u003e\n\u003ch3\u003eRefunds\u003c\/h3\u003e\n\u003cp\u003eThis is a digital product delivered immediately after purchase. By completing your order you request immediate delivery and acknowledge that, once the download has been accessed, the statutory right to cancel no longer applies to the extent permitted by law. Refund requests are handled in accordance with our published Refund Policy.\u003c\/p\u003e\n\u003cp style=\"font-size:0.85em;color:#666;\"\u003eClaude, Codex, Gemini and Cursor are trademarks of their respective owners. MCP Cart is an independent marketplace and is not affiliated with, endorsed by, or sponsored by any of them. Compatibility references describe interoperability only.\u003c\/p\u003e\n\u003c!-- mcpcart:static-blocks:end --\u003e","brand":"MCP Cart","offers":[{"title":"Default Title","offer_id":53076560544055,"sku":"MCP-NVIDIA-SKILLEVALUATOR-CREATE-CUSTOM-GRADER","price":46.99,"currency_code":"GBP","in_stock":true}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/0981\/3950\/4951\/files\/b9C4THI4DUe1GfE9d2w3K_52f72b33b5cc4be79f9e61805890cd50.jpg?v=1790852915","url":"https:\/\/mcpcart.com\/products\/create-custom-graders-with-ai-skillevaluator-byog-byot","provider":"SPF PRO","version":"1.0","type":"link"}