How-to · 17 Nov
How-to · 2 min read
How to convert web page HTML to markdown for AI models
Developers feeding web content into AI models need to clean messy page HTML into concise markdown without wasting prompt token limits. Modern web pages are flooded with non-essential elements such as navigation menus, advertising banners, cookie consent dialogs, inline CSS, and complex JavaScript bundles. When raw HTML documents are passed directly into large language model context windows, these extraneous markup tags consume valuable context space without adding meaningful information. This unnecessary bloat increases token costs, slows processing speeds, and dilutes the semantic focus of prompt contexts. Software engineers building AI workflows require a reliable automated approach to filter out web boilerplate while converting core headings, body text, tables, and hyperlinks into clean markdown.
01
Why it's hard
Extracting clean, structured markdown from arbitrary web pages presents numerous technical challenges for software development teams. Simple regex patterns or generic HTML conversion libraries frequently fail on modern dynamic websites that rely on client-side JavaScript frameworks to load content. Building custom scraping and parsing scripts for every target layout demands continuous maintenance because minor front-end layout changes break DOM traversal rules. Basic conversion tools also process entire web documents indiscriminately, retaining header links, repetitive footer lists, and tracking tags that clutter model inputs. Handling anti-bot challenges, managing browser headless instances, and maintaining session state further complicate in-house extraction pipelines.
This part describes the usual way. It makes no claims about Spicrawl, so nothing is quoted here.
02
How Spicrawl does it
Spicrawl offers a web scraping and data extraction service that transforms raw web HTML into clean, structured markdown designed for language models. The platform receives target URLs, fetches the underlying web pages, and strips away unnecessary layout elements like headers, footers, scripts, and advert blocks. When target websites require client-side script execution, the engine automatically escalates rendering to a headless browser to capture dynamic DOM elements. Developers can access the extraction capabilities through standard HTTP endpoints, command line tools, or a hosted MCP server for direct AI agent integration. Output formats include cleaned markdown, JSON data structures, or raw extracted links depending on workflow requirements.
Up to 97% fewer tokens than raw HTMLQuoted from spicrawl.com
Turn a messy web page into clean, LLM-ready textQuoted from spicrawl.com
Turn web pages into clean, usable data. Get Markdown, JSON, text, HTML, or links—ready to use.Quoted from spicrawl.com
03
Step by step
- 1Send target web page URLs to the processing endpoint through a standard REST API request, terminal CLI command, or hosted MCP server connection.
- 2Configure optional request parameters such as main content filtering, specific response formats, or automated browser escalation tiers for JavaScript sites.
- 3Allow the system to fetch the web page, execute dynamic page scripts if necessary, and parse out unwanted boilerplate elements like menus and ads.
- 4Receive clean, LLM-ready markdown text or structured JSON data to insert directly into prompt contexts or application databases without token waste.
04
Tips
Enable main content filtering parameters during API requests to automatically remove sidebars, cookie banners, and footer links from extracted documents.
Menus, footers, ads and images removed.Quoted from spicrawl.com
Configure automatic escalation modes so plain HTTP requests are attempted first, resorting to full browser rendering only when pages block initial fetches.
Start with a plain request and move up to a real browser only if the site says no.Quoted from spicrawl.com
Combine large URL lists into unified batch jobs to process multiple web pages efficiently within a single API request instead of loop calls.
batch up to 10,000 URLsQuoted from spicrawl.com
Inspect request response headers after each fetch to track credit charges and verify remaining monthly balance.
X-Credits-Charged shows what a request cost, and X-Credits-Remaining shows what is left this month.Quoted from spicrawl.com
05
Questions
How much prompt context can markdown extraction save compared to raw HTML?
Clean markdown extraction can reduce token consumption by up to 97 percent compared to raw HTML structures.
Up to 97% fewer tokens than raw HTMLQuoted from spicrawl.com
Does stripping page content cost credits if the request fails or gets blocked?
Failed requests caused by server errors, bot challenges, or refusals do not cost any credits.
Errors, bot challenges and refused requests cost 0.Quoted from spicrawl.com
How can AI agents fetch web page markdown directly through tool calls?
AI agents can connect directly to the hosted Model Context Protocol endpoint to scrape and clean web pages.
MCP endpoint: https://mcp.spicrawl.com/mcpQuoted from spicrawl.com