Any URL to LLM-Ready Markdown: A Copy-Paste Cookbook

Copy this line to your agent to turn any web page into clean, LLM-ready markdown you can chunk and embed.

set up https://monid.ai/SKILL.md and use octen /extract to turn a URL into clean markdown

To get clean, LLM-ready markdown out of any URL, send the URL to one extraction endpoint and read back the article text with the nav bars, cookie banners, and ad rails already stripped. That single call replaces the headless browser plus boilerplate-remover you would otherwise build and babysit, and it batches up to 20 pages at once. This is a content extraction job, turning a page you already found into text a model can read, which is a different problem from searching or discovering pages in the first place.

TL;DR

• One call to octen /extract on Monid turns a URL into clean markdown, ready to chunk and embed. Send up to 20 URLs in a single request.
• The query field switches the return from the whole page to only the passages relevant to your intent, which cuts tokens before they ever hit your embedder.
• Failed URLs come back with status: "failed" and are not billed, so a batch with a few dead links only charges for the pages that extracted.
• The DIY alternative is a headless browser plus a readability pass you maintain against every site redesign. Extraction bills per result, a fraction of a cent per page, shown before you run. See monid.ai/tools.

Why not just fetch the HTML yourself

The obvious move is fetch(url) plus an HTML parser. It works on a clean blog and falls apart everywhere else. Modern pages render their body with JavaScript, so a plain fetch returns an empty shell and you reach for a headless browser like Playwright. Now you own a browser install in your deploy image, a per-page timeout budget, and a memory footprint that spikes under concurrency.

Then you still have raw DOM, not article text. You bolt on a boilerplate stripper such as Mozilla's Readability to drop the header, sidebar, comment thread, and newsletter modal. Readability is good, but it is heuristic, so it guesses wrong on paywalls, single-page apps, and anything with an unusual layout, and every large site redesign is a fresh round of debugging. The extraction endpoint absorbs all of that: rendering, stripping, and markdown conversion happen server-side, and you get back text.

STEP 1: install and add a key

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More details in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add --label main --key <your-api-key>

More details in the CLI quickstart.

STEP 2: read the schema for free

Before you spend anything, inspect the endpoint. Discovery and inspection never bill, so this is where you confirm the fields and the exact per-result price.

monid discover -q "web page to markdown"
monid inspect -p octen -e /extract

The body takes urls (an array of 1 to 20, required), query (optional, returns intent-focused highlights), format (defaults to markdown, or text), and max_age_seconds (how stale a cached copy may be before a re-fetch). Results come back in the same order you sent them, which matters once you batch.

STEP 3: one URL to markdown

Start with a single page. This is the whole job in one line: the -w flag waits inline and prints the result.

monid run -p octen -e /extract -i '{"urls":["https://www.paulgraham.com/greatwork.html"]}' -w

You get back a record for that URL with its cleaned markdown, the detected page structure, and a status of success. The markdown keeps the heading hierarchy, which is the field that matters most downstream, because those headings are your natural chunk boundaries. Feed the body straight to a splitter that breaks on ## and ### and you get chunks that respect the document's own sections instead of arbitrary character windows.

STEP 4: batch up to 20 URLs in one call

A crawl frontier, a sitemap slice, or a reading list is the same call with a longer array. Send up to 20 URLs and read back one record per URL, in input order.

monid run -p octen -e /extract -i '{
  "urls":[
    "https://www.paulgraham.com/greatwork.html",
    "https://www.paulgraham.com/ds.html",
    "https://www.paulgraham.com/genius.html"
  ]
}' -w

The batch behavior that saves money: any URL that fails to extract comes back with status: "failed" and an error_message, and those failures are free. A dead link, a hard paywall, or a timeout in your list of 20 does not cost you anything, so you pay only for the pages that actually produced markdown. Loop your full URL set in chunks of 20 and the arithmetic stays simple.

STEP 5: return only what the query needs

Full-page markdown is right when you are archiving or indexing the whole document. When you already know the question, the query field changes the return to only the passages relevant to that intent, so you strip the noise before it ever reaches your embedder.

monid run -p octen -e /extract -i '{
  "urls":["https://www.paulgraham.com/greatwork.html"],
  "query":"how to choose what to work on"
}' -w

This is a deliberate cost and quality lever, not just a convenience. Fewer tokens in means a smaller embedding bill, a tighter index, and less irrelevant context diluting a retrieval answer. Use max_age_seconds alongside it to control freshness: a large value serves a recent cached copy cheaply for stable pages, a small value forces a re-fetch for anything that changes hour to hour.

The cost tally, in magnitudes

Reason about the shape of the bill, not a price sheet. Extraction meters per successfully extracted URL at a fraction of a cent each. So a batch of 20 pages, if all 20 extract, is still comfortably under a cent of magnitude, and a few thousand pages lands in the low single-digit dollars. Failed URLs subtract themselves from that total because they do not bill.

Put that next to the DIY line item. The build is not the browser code, it is the maintenance: the on-call page when a site redesign breaks your selectors, the memory tuning when concurrency spikes, the readability edge cases you patch by hand. Those are engineering hours, the most expensive input you have. The per-result fee buys them back. Live per-endpoint pricing, including the cheaper and pricier extraction alternatives on the catalog, is at monid.ai/tools.

Where this fits with search

Extraction is the read step, not the find step. If your agent still needs to discover which URLs to pull, that is a web search job, and a search API returns the candidate links (some even return page text inline). Once you have the URLs, /extract is what turns them into markdown at scale, on the same wallet. Monid is a pay-per-call data API marketplace: one interface and one balance to discover, inspect, and run hundreds of external data endpoints without a separate signup per vendor, shipping as an MCP server so an agent can drive the whole loop itself. Only the run step bills; discovery and inspection are free.

FAQ

What does /extract return for one URL? A record with the page's cleaned markdown (or plain text if you set format to text), the detected structure, and a status. The markdown preserves headings, so you can chunk on section boundaries and embed directly.

How many URLs can I send at once? One to 20 per call. Results come back in the same order you sent them, and any URL that fails extraction returns status: "failed" with an error message and is not billed.

What does the query field actually change? Without it you get the whole page as markdown. With it you get only the passages relevant to your query, which trims tokens before embedding and sharpens retrieval. It is the right choice when you already know the question you are answering.

How much does it cost? Per successfully extracted URL, a fraction of a cent, always shown on the free inspect step before you run. Failed URLs are free. Current prices, and the pricier full-render alternatives, are at monid.ai/tools.

Do I need to run a headless browser myself? No. Rendering, boilerplate stripping, and markdown conversion happen server-side. After you add a key an agent can run the whole extraction loop through the MCP server. Start at monid.ai.