Scrape URL to Markdown
Any public page as clean, LLM-ready markdown. Nav, scripts and footers stripped; links and structure preserved. Optional CSS selector to scope the extraction.
POST https://api.bottrunk.com/s/scrape-markdown
Call it
# once, on the machine your agent runs on npx bottrunk-mcp wallet # prints the address; send it 0.3 ALGO, then USDC claude mcp add bottrunk -- npx -y bottrunk-mcp # then ask your agent for it — the tool is `bottrunk_scrape_markdown`
# Ask. You get a 402 with the payment requirements.
curl https://api.bottrunk.com/s/scrape-markdown \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com/pricing"}'
# Pay and retry. Any x402 client does this for you.
curl https://api.bottrunk.com/s/scrape-markdown \
-H "PAYMENT-SIGNATURE: <signed USDC transfer>" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com/pricing"}'
from x402.clients import requests as x402_requests
session = x402_requests.x402_http_adapter(account) # your agent's Algorand account
r = session.post("https://api.bottrunk.com/s/scrape-markdown", json={"url":"https://example.com/pricing"})
print(r.json())
print(r.headers["PAYMENT-RESPONSE"]) # settlement receipt
import { wrapFetchWithPayment } from "@x402-avm/fetch";
const r = await fetchWithPay("https://api.bottrunk.com/s/scrape-markdown", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({"url":"https://example.com/pricing"}),
});
console.log(await r.json());
How it behaves
The decisions this service makes on your behalf, and the ones that end in a refusal. A 4xx is never settled, so a refusal costs you nothing.
| What is stripped | script, style, noscript, nav, footer, header, aside, form, iframe, svg, template — removed before conversion, so their text never reaches you. |
| What is kept | Headings, paragraphs, lists, tables, links and emphasis, as GitHub-flavored markdown. Runs of blank lines are collapsed. |
| Where it reads from | Your selector if you give one. Otherwise the first of main, article, body. |
| A selector that matches nothing | Refused with 422 and not charged, rather than quietly returning the whole page. A malformed selector is refused the same way. |
| Non-HTML URLs | A URL that answers with JSON, a PDF or an image is refused with 422. A server that sends no content-type at all is scraped anyway. |
| No JavaScript | The page is fetched, not rendered. A site that builds its content client-side returns almost nothing. render_js is accepted and ignored. |
| Size and time | Pages over 2 MB are refused with page too large. Connect times out at 5 s, the whole fetch at 20 s, and at most 3 redirects are followed. |
| Who we look like | Requests go out as BotTrunk/0.1 (+https://bottrunk.com/docs). Sites that block unknown agents will block this one. |
| Private addresses | Hostnames that resolve to private, loopback or link-local space are refused with 422 before any request is made. Do not point this at localhost or an intranet. |
| Method | POST only. A GET on the same path returns this page. |
Input
| url | string | Public http(s) URL to fetch. |
| render_js | boolean | Reserved: headless rendering is not available yet; the flag is accepted and ignored. |
| selector | string | Optional CSS selector to scope the extraction. |
Output
| markdown | string | Body content as GitHub-flavored markdown. |
| title | string | Document title. |
| word_count | integer | Words in markdown, for budgeting tokens. |