Crawl4AI
Verifiedby UncleCode
Crawls and scrapes web pages into clean Markdown for RAG pipelines, with CSS, XPath, and LLM-driven structured extraction, deep crawling, and full browser session control.
- Access
- APICLIMCP
- Auth
- None
- Skills
- 5
Skills
LLM Ready Markdown
Generates clean Markdown with heuristic and BM25 noise filtering, numbered citation references for links, and pluggable strategies.
Structured Data Extraction
Extracts JSON matching a user-defined schema using CSS or XPath selectors, or delegates to an LLM for semantic field extraction.
Deep Site Crawling
Traverses whole sites with BFS and best-first strategies under page limits, using prefetch URL discovery and resumable crash recovery.
Browser Session Control
Drives Chromium, Firefox, or WebKit with persistent profiles, proxies, stealth mode, hooks, and reusable sessions for multi-step crawls.
Docker API Server
Exposes a FastAPI /crawl endpoint with JWT authentication plus MCP integration so agent clients can crawl over HTTP.
Related Agents
context.dev
by context.dev
Scrapes, crawls and searches the web for agents through a REST API, SDKs, CLI and hosted MCP server, returning…
AgentQL MCP
by TinyFish
Extracts structured web data using natural language queries — no XPath or selectors needed. Works via Python/JS SDKs…
Browser Use
by Browser Use
Open-source Python library for AI browser automation. Control any website with natural language; cloud adds stealth…
Moli
by Lexmount
Fetches, extracts, and automates web pages in an open-source Rust headless browser that outputs Markdown or semantic…