Agent Crawl Readiness

Name: Agent Crawl Readiness MCP Server
Author: sadri-dridi

remote

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

Getting Started

npx @getmcp/cli add io.github.sadri-dridi/agent-crawl-readiness

Requires Node.js 18+. Community-contributed server. Review source before installing.

Claude Desktopguide →Vscodeguide →Cursorguide →Windsurfguide →

Configuration

Config file: ~/Library/Application Support/Claude/claude_desktop_config.json

JSON

{
  "mcpServers": {
    "io.github.sadri-dridi/agent-crawl-readiness": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp",
      "transport": "streamable-http"
    }
  }
}

Claude Desktop MCP documentation (opens in new tab)

Authorsadri-dridi

Version1.0.0

TransportRemote

Updated15h ago

github.com (opens in new tab)agent-ads-production-9f45.up.railway.app (opens in new tab)