Firecrawl is an open-source API that turns any web page into clean Markdown, ready for a language model, so your AI agent can read the web without drowning in HTML. An agent that only reasons over its memory is stuck in the past; to check prices, documentation or news it needs a tool that brings it reliable web data. In this guide you will see what Firecrawl is, how its scrape, crawl and extract endpoints work, how to self-host it with Docker on your own machine, how to hand it to an agent as a tool and when it beats a hand-rolled scraper. The same explanation is available in Spanish.

Key takeaways

  • Firecrawl is an open-source API (AGPL-3.0 licence, SDKs under MIT) that turns web pages into Markdown, HTML, screenshots or structured JSON; it has over 145,000 stars on GitHub and is on version v2.11.0, released on 19 June 2026.
  • It exposes five operations: scrape (one URL), crawl (a whole site), map (discover URLs), search (search the web) and extract (schema-guided structured data).
  • It returns clean Markdown by default: it strips menus, ads and noise, exactly what fits into a model’s context window without wasting tokens.
  • It is self-hostable with Docker: git clone the repo, docker compose up and the API runs at http://localhost:3002; it asks for around 8 GB of RAM because it spins up a Playwright browser, Redis and a database.
  • It ships official SDKs for Python (firecrawl-py) and Node (firecrawl), an MCP server and a CLI, so it slots into both your own MCP server and any agent framework.

What is Firecrawl?

Firecrawl is an API that takes a URL and returns its content in a format a language model understands effortlessly. The official documentation describes it as "the context API to search, scrape and interact with the web at scale". The underlying idea is simple: the HTML of a modern site is full of menus, banners, scripts and markup that add nothing for the model and burn tokens; Firecrawl renders it (JavaScript included), cleans it up and hands you ready-to-use Markdown.

The project is free software under the AGPL-3.0 licence, while its client libraries ship under the more permissive MIT. It has had an enormous reception: it holds more than 145,000 stars on GitHub and is on version v2.11.0, released on 19 June 2026. Behind it is the company Firecrawl (formerly Mendable), and in 2026 the GitHub organisation moved from mendableai to firecrawl, so some older tutorials still cite the old path.

You can use it two ways: against the Firecrawl cloud (no key to get started, a free account raises the limits) or hosted on your own machine. To try it from Python you need just two lines:

pip install firecrawl-py

from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key="fc-YOUR_KEY")

doc = firecrawl.scrape(
    "https://en.wikipedia.org/wiki/Intelligent_agent",
    formats=["markdown"],
)
print(doc.markdown)

What formats does it offer: scrape, crawl and extraction to Markdown?

The flagship endpoint is /scrape: you pass it a URL and choose the output formats. By default it returns markdown, but it also accepts html, rawHtml, links (every link on the page), screenshot (an image) and json (schema-guided structured extraction). You can request several at once and decide what your agent needs.

When one page is not enough, /crawl walks a whole site: it discovers the URLs, follows pagination, renders the JavaScript and returns every page as Markdown within a single asynchronous job. For more surgical tasks there are /map (quickly list every URL on a domain) and /search, which searches the web and returns the content of the results in one step, handy for giving an agent search ability without standing up an engine like SearXNG.

The gem for agents is /extract: instead of loose text, you pass a JSON schema and Firecrawl fills the fields by reading one or more pages. That turns a product sheet or a directory into typed data your program consumes directly, without writing CSS selectors or brittle regular expressions.

How to self-host Firecrawl with Docker?

Firecrawl can be hosted on your own machine, valuable when you handle sensitive data or want to avoid per-request costs. The repository includes a docker-compose that spins up the API, a Playwright browser for rendering, Redis for the job queue and a database. The process is three steps:

git clone https://github.com/firecrawl/firecrawl.git
cd firecrawl
cp .env.example .env

docker compose build
docker compose up -d

In the .env file it is worth setting PORT=3002, HOST=0.0.0.0 and USE_DB_AUTHENTICATION=false for a simple local install. Once up, the API answers at http://localhost:3002 and you can fire your first scrape with curl:

curl -X POST http://localhost:3002/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://firecrawl.dev", "formats": ["markdown"]}'

Two important warnings. First: budget memory, because the headless browser is greedy and the documentation itself recommends around 8 GB of RAM and about 20 GB of disk. Second: the self-hosted version does not include Fire-engine, the proprietary engine that handles IP rotation, proxies and anti-bot evasion. For normal scraping it is more than enough; if your target blocks aggressively, the paid cloud performs better there.

How to give your agent a web tool?

An agent is a model inside a loop that can call tools; giving it web access is as simple as wrapping a Firecrawl call in a function and registering it as a tool. The pattern is identical whether you use OpenAI’s function calling, Anthropic’s tools or a framework like LangChain:

from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key="fc-YOUR_KEY")

def read_page(url: str) -> str:
    """Fetch a page and return it as clean Markdown."""
    doc = firecrawl.scrape(url, formats=["markdown"])
    return doc.markdown

Now the agent decides for itself when to call read_page: if the user asks about the content of a link or a recent piece of news, the model emits a tool call, you run the function and hand back the Markdown as an observation. That clean text integrates into the context far better than raw HTML, so the model hallucinates less and cites better. If what you want is for the agent to navigate and interact with the page (fill in forms, click buttons), the complementary piece is a browser agent like Browser Use; Firecrawl shines when you only need to read the web, not operate it. And if you juggle many integrations at once, platforms like Composio offer Firecrawl as one more ready-to-connect tool.

Firecrawl or a scraper of your own?

The temptation to write your own scraper with requests and BeautifulSoup is real, and for a static, stable page it can be enough. The trouble shows up with the modern web: content loaded with JavaScript, structures that change constantly, endless pagination, rate limits and converting to decent Markdown. Each of those fronts is maintenance that piles up.

Firecrawl lifts exactly that load: it renders the JavaScript, normalises the output to Markdown, manages the queue and retries, and exposes a uniform schema for every site. In exchange you depend on a service (or on maintaining your own Docker instance) and, in the cloud, you pay per request. The rule of thumb: for a one-off scrape of a simple site, your own script pays off; to feed an agent that reads dozens of different, changing sites in production, Firecrawl saves you weeks of patches. Being open source, you also avoid vendor lock-in: if it stops convincing you tomorrow, you migrate to your self-hosted instance.

Frequently asked questions

Is Firecrawl free and open source?

Firecrawl’s core is open source under the AGPL-3.0 licence and its SDKs under MIT, so you can read the code and host it for free on your own machine. Separately there is a cloud service with a free plan to get started and paid plans by volume. In other words: the software is free, but the managed infrastructure (with Fire-engine, proxies and scaling) is a commercial product.

Can I host Firecrawl on my own server?

Yes. The repository ships a docker-compose that spins up the API, the rendering browser, Redis and the database with docker compose up, and it listens on port 3002. Keep two things in mind: it needs around 8 GB of RAM and the self-hosted version does not include Fire-engine, the engine that gets past blocks and banned IPs. For standard scraping it is more than enough.

How does scrape differ from crawl?

scrape works on a single URL and returns its content in the format you request. crawl starts from a URL, discovers and follows the site’s internal links, and scrapes many pages in a single asynchronous job. Use scrape when you know exactly which page you want and crawl when you need to dump a whole section, such as an entire documentation set, to index it.

Conclusion

Firecrawl solves a very concrete problem for AI agents: giving them eyes on the web without choking on HTML. With scrape, crawl, map, search and extract it covers everything from a single page to a whole site, always as clean Markdown; with its SDKs, its MCP server and its docker compose it integrates in minutes, in the cloud or on your own machine. Remember that the self-hosted version drops Fire-engine and asks for around 8 GB of RAM. The next step is to bring it up with docker compose up, wrap a scrape call in a function and register it as your agent’s tool.

Sources

  1. Official Firecrawl documentation
  2. Firecrawl on GitHub
  3. SELF_HOST self-hosting guide
  4. Independent Firecrawl review at CoddyKit

Route: Agent Ecosystem: MCP, Gateways and Platforms