Agent Crawl Readiness
Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.
Servidor MCP · de sadri-dridi
Resumen
Agent Crawl Readiness es un servidor MCP publicado por sadri-dridi. Marketplaces que leemos y que lo publican: 1 (Official MCP Registry). Lo vimos por primera vez el 2026-10-11 y desde entonces no hemos registrado cambios. Según nuestra pasada por los marketplaces del 2026-10-16.
Qué pide
- Se ejecuta en remoto
Ninguno de los marketplaces que leemos publica sus herramientas ni los ajustes que pide.
Lo que declara, pide y ejecuta, completo
Instala solo plugins de desarrolladores en los que confíes. Mostramos lo que publican los marketplaces y no probamos plugins.
Funciona con
- Claude Code
- Codex
- Cursor
- VS Code
- Copilot CLI
Un enlace de un clic abre el cliente, que pide confirmación antes de añadir el servidor.
Según los marketplaces que lo listan y el transporte MCP que declara, tal como lo documenta cada cliente. No probamos plugins.
Conectar
Comandos y configuraciones generados solo a partir de lo que declaran sus marketplaces. No los ejecutamos por ti ni probamos plugins.
Como servidor MCP remoto
Generado a partir de lo que declara Official MCP Registry.
claude mcp add --transport http agent-crawl-readiness https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp[mcp_servers.agent-crawl-readiness]
url = "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"{
"mcpServers": {
"agent-crawl-readiness": {
"url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
}
}
}Dónde está publicado
| Marketplace | Estado | Versión | Cifras | Verificado | Visto por primera vez |
|---|---|---|---|---|---|
| Official MCP Registry (Abrir el registro) | Publicado | 1.0.0 | Forks (GitHub): 0 Estrellas (GitHub): 0 |
Sin marca |
Cada cifra es del propio marketplace y nunca se suma entre marketplaces. Un marketplace que no publica cifras no muestra ninguna: sin número no es cero.
Qué cambió
Nada ha cambiado en este plugin desde que lo seguimos.
Plugins similares
Productos cuyos propios nombres, descripciones, herramientas y skills son los más cercanos a este por su significado, en todos los marketplaces que leemos. La cercanía se ordena solo dentro de esta lista.
-
Analyse robots.txt content and report whether a path may be crawled.
-
ai-crawler-readiness com.lumnika/ai-crawler-readiness
Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
-
crawl-readiness io.github.Grubbomatic/crawl-readiness
Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.
-
ActableSite AI Crawler Monitor
Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
Live web checks for AI agents: sitemaps, robots.txt, URL status, broken links, feeds, citations.
-
AI visibility checker: GPTBot vs OAI-SearchBot, robots.txt audit, raw-HTML schema. 165 crawlers.
-
robots-txt-ai-crawler-audit io.github.jmshinhwa/robots-txt-ai-crawler-audit
Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI
-
ai-visibility com.lastminutedealshq/ai-visibility
AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.
-
agent-ready-kit com.mudko/agent-ready-kit
Agent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.
-
Can GPTBot, ClaudeBot and PerplexityBot reach a site? Real fetch per crawler, firewall/CDN blocks.
Plugins similares por marketplace
La misma pregunta a cada marketplace por separado: ¿hay allí un equivalente de este plugin?
Publicado en Claude Code: claude-plugins-community (5)
-
A Claude Code plugin that gives Claude direct access to the Browserless.io REST APIs: scrape webpages, take screenshots, generate PDFs, search the web, map site structures, and…
-
Audits public-facing websites across accessibility, SEO, GEO (AI search readiness), and usability -- evaluates 17 checks, assigns a letter grade, and generates a client-ready PDF…
-
Skyvern browser automation skills for web scraping and interaction via remote MCP.
-
Data freshness and endpoint liveness verification for AI agents. Probes external APIs and endpoints before agents act on their data, returning deterministic FRESH/STALE/UNKNOWN…
-
Web scraping skills for Claude Code powered by the Zyte API — scrape sites, generate and run Scrapy spiders, define extraction schemas, and ship to Scrapy Cloud.
Publicado en Claude Code: claude-plugins-official (5)
-
Web scraping skills for Claude Code powered by the Zyte API — scrape sites, generate and run Scrapy spiders, define extraction schemas, and ship to Scrapy Cloud.
-
Web scraping, Google search, structured data extraction, and MCP server integration powered by Bright Data. Includes 7 skills: scrape any webpage as markdown (with bot…
-
playwright-mcp io.github.microsoft/playwright-mcp
Playwright Tools for MCP
-
Give Claude a real browser — your Chrome or a Browser Use Cloud browser. Use it whenever a task involves a website or web app: browsing, scraping and data extraction, filling…
-
Web scraping and crawling powered by Firecrawl. Turn any website into clean, LLM-ready markdown or structured data. Scrape single pages, crawl entire sites, search the web, and…
Publicado en Claude: knowledge-work-plugins (5)
-
Web scraping, Google search, structured data extraction, and MCP server integration powered by Bright Data. Includes 7 skills: scrape any webpage as markdown (with bot…
-
The complete web toolkit for your agent. Search the web and get answers in milliseconds. Fetch any URL and get clean markdown content back. Send a browser agent to navigate sites…
-
Give Claude a real browser — your Chrome or a Browser Use Cloud browser. Use it whenever a task involves a website or web app: browsing, scraping and data extraction, filling…
-
Exa AI web search, deep research, and content extraction. Provides MCP tools and research skills for comprehensive web search, people discovery, company research, academic papers…
-
Nimble web data toolkit — search, extract, map, crawl the web and work with structured data agents
Publicado en Codex: openai-curated (openai/plugins) (5)
-
Cloudflare platform plugin with curated skills for Workers, Wrangler, and Agents SDK plus the official Cloudflare API MCP server.
-
Codex Security workflows for security scans, analysis, and investigation.
-
Evaluate Codex skills and plugins from chat with a beginner-friendly start command, local-first reports, token budget explanations, and guided benchmarking.
-
Work with SharePoint using the configured Microsoft SharePoint app connector.
-
CrowdStrike Falcon Fusion skills for authoring, deploying, and executing Fusion workflows. Includes live action discovery, YAML authoring with schema validation, workflow import…
Publicado en Community GitHub marketplaces (5)
-
SEO audits and AI crawler checks. Honest findings only; no invented scores.
-
Browser automation and web page analysis skills using playwright-cli: connect via CDP, probe bot protection, dismiss overlays, capture DOM trees, reduce pages to skeletons…
-
A Claude Code plugin that gives Claude direct access to the Browserless.io REST APIs: scrape webpages, take screenshots, generate PDFs, search the web, map site structures, and…
-
Scrape, crawl, map, search, parse, extract, and change-track the web with fastCRW, the open-source, self-hostable Firecrawl alternative. Single Rust binary, ~6.6 MB idle RAM…
-
AI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.
Publicado en Copilot CLI: github/awesome-copilot (5)
-
Helps devs extract data from any website with thousands of trusted scrapers, crawlers, and automations from Apify Store. Integrate ready-made Actors to access all popular social…
-
MCP server for Chrome DevTools
-
Safe BLE diagnostics, evidence workflows, and guarded automation for AI agents.
-
Three-layer verification pipeline for AI output. Extracts claims, finds sources, and flags hallucination risks so humans can verify before acting.
-
Drive Microsoft AgentRC from Copilot chat: assess AI readiness, generate Copilot instructions (flat or nested with applyTo globs for monorepos), and manage policies. Produces a…
Publicado en Copilot CLI: github/copilot-plugins (5)
-
Toolkit for building Microsoft 365 Copilot declarative agents — scaffolding, JSON manifest development, and capability configuration.
-
Advanced Security plugin for GitHub Copilot.
-
mcp-apps
Generate MCP App widgets for MCP tools. Produces self-contained HTML widgets using the MCP Apps protocol.
-
workiq
Query Microsoft 365 data with natural language — emails, meetings, documents, Teams messages, and more.
-
Build Power Apps Canvas Apps using the Canvas Authoring MCP server.
Publicado en Cursor: cursor/plugins (5)
-
Browse and automate the web with TinyFish: run multi-step web automations, search the web, extract clean page content, and drive remote stealth browser sessions.
-
playwright-mcp io.github.microsoft/playwright-mcp
Playwright Tools for MCP
-
CLI-backed repo compatibility scans plus agents that audit startup, validation, and docs against reality.
-
Search docs, read pages, and update tables.
-
Scaffold and validate new agent plugins.
Publicado en Docker MCP Catalog (5)
-
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
-
oxylabs-mcp io.oxylabs/oxylabs-mcp
Fetch and process content from specified URLs & sources using the Oxylabs Web Scraper API.
-
scrapegraph-mcp io.github.ScrapeGraphAI/scrapegraph-mcp
AI-powered web scraping and data extraction capabilities through ScrapeGraph API
-
Apify is the world's largest marketplace of tools for web scraping, data extraction, and web automation. You can extract structured data from social media, e-commerce, search…
-
mcp-server-browserbase io.github.browserbase/mcp-server-browserbase
MCP server for AI web browser automation using Browserbase and Stagehand
Publicado en Official MCP Registry (5)
-
Analyse robots.txt content and report whether a path may be crawled.
-
ai-crawler-readiness com.lumnika/ai-crawler-readiness
Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
-
crawl-readiness io.github.Grubbomatic/crawl-readiness
Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.
-
ActableSite AI Crawler Monitor
Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
-
Live web checks for AI agents: sitemaps, robots.txt, URL status, broken links, feeds, citations.
Encontrados por significado, no por código ni repositorio compartidos: los plugins similares siguen siendo productos distintos, y esto no es una reseña ni una recomendación.
¿Quieres esta marca para tu plugin en nuestro catálogo?
Demuestra que lo mantienes con un archivo en su repositorio, un archivo en tu dominio o un registro DNS, y podrás mostrar esta marca en la ficha, con banners en vivo para tu web y tu README. Cómo funciona la verificación
Además recibirás avisos cuando un marketplace lo publique, lo retire, cambie su marca verificada o lo que pide, y su endpoint remoto quedará monitorizado como un servicio web.
Datos de Official MCP Registry. Las cifras proceden de nuestros propios recorridos periódicos por los marketplaces, leída por última vez el . No publicamos datos de contacto de los editores de plugins.