Master Docs with AI: Complete Site Crawling & Markdown Conversion

Master Docs with AI: Complete Site Crawling & Markdown Conversion

This skill is designed to crawl an entire documentation or design-system website, converting it into a single, LLM-ready Markdown corpus. It discovers every page from the site's sitemap.xml, or follows links if no sitemap is available, extracts the main content in Markdown format, and downloads images locally. It also renders JavaScript-heavy pages using a headless browser.

What this skill does

• Crawls entire websites, such as documentation, design-system, API reference, component library, or knowledge base sites.
• Utilizes the site's sitemap.xml for page discovery, with a same-origin link-following fallback.
• Extracts each page's main content into clean Markdown.
• Downloads associated images, including inline base64 images, to a local folder.
• Handles JavaScript-heavy pages using a headless browser for accurate content capture.

Who it is for

• Developers and teams using AI coding agents such as Claude Code, Cursor, and Codex.
• Professionals involved in documentation archiving, research, or conversion for AI model training.
• Researchers analyzing design systems or knowledge bases for contextual insights.

Use cases

• Crawling a multi-page documentation site to create a comprehensive Markdown repository.
• Converting a design-system or API reference site into a format suitable for ingestion by large language models (LLMs).
• Archiving an entire component library or knowledge base for offline use or research.

Technical details

• Employs agent-skills for page crawling and Markdown conversion.
• Capable of rendering complex sites using a headless browser to ensure accurate page content capture.
• Outputs clean, LLM-compatible Markdown with relative image paths for local referencing.

Note: This skill targets publicly available information only. Users must comply with the target site's terms of service and it does not bypass any authentication mechanisms or CAPTCHA.

Source & Licence

This package is built on open-source work published by CaesiumY (CaesiumY/ko-design-md) and distributed under MIT. The original licence text and copyright notice are included in your download.

Personal and commercial use, modification and redistribution are permitted, provided the original copyright and licence notice are retained.

Your purchase covers curation, licence verification, packaging, documentation and instant delivery. It does not grant exclusive rights to the underlying open-source code, which remains available under its original licence.

Delivery & Support

• Delivery: instant — a secure download link is emailed to you as soon as payment is confirmed.
• Format: ZIP archive containing the skill files, documentation and the original licence.
• Support: support@mcpcart.com — we aim to reply within 2 business days.
• Updates: updates are included only where stated on this page.

Refunds

This is a digital product delivered immediately after purchase. By completing your order you request immediate delivery and acknowledge that, once the download has been accessed, the statutory right to cancel no longer applies to the extent permitted by law. Refund requests are handled in accordance with our published Refund Policy.

Claude, Codex, Gemini and Cursor are trademarks of their respective owners. MCP Cart is an independent marketplace and is not affiliated with, endorsed by, or sponsored by any of them. Compatibility references describe interoperability only.