Let’s talk about the Gemini tool nobody’s using: URL context. It’s a built-in web scraper in the Gemini API that can parse HTML, PDFs, XML, JSON, CSV, and images – all multimodal, all free (you only pay for the tokens the model processes), and all without a single external service dependency.
No reader APIs. No crawl services. No headless browsers. Just provide a URL, and Gemini grounds its answers in the live content from that page.
I’ve tested this against dedicated scraping tools like Firecrawl and Crawl4AI, and here’s the reality: for most use cases, Gemini’s URL context is faster, more reliable, and cheaper. Yet it’s probably the least-used tool in the Gemini API. Let’s fix that.
How URL Context Actually Works
The URL context tool follows a two-step approach that balances speed, cost, and data freshness:
-
Check Indexed Cache – Since Google is one of the world’s largest search engines, when you provide a URL, Gemini first looks at Google’s indexed cache. If the content is already cached, it pulls from there instantly.
-
Live Fetch Fallback – If the URL isn’t cached or the content has changed recently, Gemini falls back to live fetching – it scrapes the page in real-time.
This hybrid approach means you get cached performance for static content (documentation, archived pages) and fresh data for dynamic content (news articles, live dashboards). And critically, you’re not paying for a separate scraping service; you’re just paying for the input tokens that result from the URL content being processed.
Initially, URL context only supported text extraction from HTML. But as of late 2025, Google added support for:
– Images (PNG, JPEG, BMP, WebP)
– PDFs with visual understanding (not markdown conversion – actual page-level visual parsing)
– Structured data (JSON, XML, CSV)
That PDF capability is the sleeper feature. Gemini doesn’t convert PDFs to markdown and then parse the text. It looks at every page visually, which means it understands tables, diagrams, charts, and layout context that pure text extraction misses.
The Limits (And Why They Don’t Matter)
Let’s address the constraints:
– 20 URLs per request – Hard limit
– 34MB per URL – Content size cap
– Publicly accessible URLs only – No localhost, private networks, or tunneling services like ngrok
– No paywalled content – Can’t access subscription-required pages or platforms like YouTube videos
For 90% of web scraping use cases – documentation analysis, competitive research, content aggregation, data extraction – these limits are non-issues. If you need to scrape 100 URLs, batch them into five requests. If a URL exceeds 34MB, you’re dealing with an edge case (massive PDFs or media-heavy pages).
The real limitation is authentication. If your target content requires login credentials, URL context won’t work. But for public web data – which is what most scraping targets anyway – you’re clear.
Comparing URL Context to Dedicated Scrapers
Let’s be direct: there are excellent dedicated scraping tools. Firecrawl’s reader API, Crawl4AI (open-source), and commercial services like ScrapingBee all have their place. But URL context has three advantages:
1. Zero External Dependencies
You’re already using the Gemini API. No additional service to configure, no separate API keys, no extra billing to track. It’s baked into the SDK.
2. Multimodal Out of the Box
Dedicated HTML scrapers excel at text extraction, but handling PDFs and images usually requires separate libraries or services. URL context processes all three natively with Gemini’s vision capabilities.
3. Cost Structure
You’re not paying per scrape; you’re paying per token processed. If a URL returns 10,000 tokens of content and you’re using Gemini 3 Flash, that’s ~$0.00125 at current pricing (based on Gemini 3 Flash input cost of ~$0.125 per million tokens). Compare that to commercial scraping APIs charging $0.001-0.01 per page.
For high-volume scraping, dedicated tools with caching and rate limit handling still win. But for ad hoc research, document analysis, and RAG (retrieval-augmented generation) pipelines, URL context is often faster and simpler.
The PDF Use Case: Why Visual Understanding Matters
Here’s where URL context shines: PDF parsing. Traditional scrapers convert PDFs to markdown using tools like pypdf or pdfplumber, which works fine for text-heavy documents but breaks on complex layouts.
Gemini’s document understanding capabilities mean it processes PDFs visually – it “sees” each page as an image and interprets tables, charts, and multi-column layouts in context. I tested this on the “Attention Is All You Need” paper (the Transformer architecture paper) by providing the PDF URL and asking about Figure 1. Gemini accurately described the multi-head attention diagram, including the specific components and data flow.
That kind of visual comprehension is something markdown-based parsers fundamentally can’t deliver. If your use case involves technical papers, financial reports, or any PDF with non-linear layouts, URL context is a game-changer.
Setting It Up: API Structure
Using URL context requires the latest Gemini SDK (genai Python package version 0.8.0+). The setup is straightforward. First, configure your tools:
tools = [
{
"url_context": {}
}
]
Then pass the tools config to your model call:
import google.generativeai as genai
genai.configure(api_key="YOUR_API_KEY")
model = genai.GenerativeModel('gemini-3-flash-preview')
response = model.generate_content(
"Extract the key features from https://example.com/product-specs",
tools=tools
)
That’s it. Gemini will detect the URL in your prompt, invoke the URL context tool automatically, scrape the content, and ground its response in that data.
You can also use it via Google AI Studio with a simple toggle – enable “URL context” in the tool settings, provide a URL in your prompt, and the model handles the rest. For production use, the REST API is more stable and avoids SDK version dependencies (highly recommended if you’re building an application).
Combining URL Context with Google Search
Here’s where it gets powerful: you can combine URL context with Gemini’s built-in Google Search tool. This creates a two-step agentic workflow:
- Search for relevant URLs on a topic
- Scrape those URLs and synthesize the content
For example:
tools = [
{"google_search": {}},
{"url_context": {}}
]
response = model.generate_content(
"Find the latest benchmarks for GPT-5.2 vs Claude Sonnet 4.5 and summarize the results",
tools=tools
)
Gemini will:
1. Use Google Search to find recent articles with benchmark data
2. Use URL context to scrape those pages
3. Generate a summary with citations
That’s a deep research workflow without any external scraping infrastructure. You’re not manually finding sources, copying content, and feeding it to the model. The model is doing all three steps autonomously.
Practical Applications I’ve Tested
I’ve been using URL context for a few months, and here are the use cases where it outperforms alternatives:
1. Competitive Intelligence
Scrape competitor product pages, extract pricing tables, feature lists, and changelog data. Gemini’s vision capabilities handle complex HTML layouts and dynamically generated content.
2. Documentation Analysis
Feed API documentation URLs and ask Gemini to generate client library code or explain deprecated endpoints. Works especially well for OpenAPI specs and technical guides.
3. PDF Research
Provide academic paper URLs (arXiv, IEEE Xplore) and ask specific questions about methodology, results, or figures. The visual PDF understanding catches details that text extraction misses.
4. News Aggregation
Scrape multiple news articles on a topic and ask Gemini to identify consensus views, outlier takes, and factual conflicts. Great for rapid synthesis.
5. Structured Data Extraction
Point Gemini at public datasets (JSON APIs, CSV files) and ask it to clean, transform, or analyze the data. No pandas required.
Why Developers Aren’t Using This
URL context launched quietly in mid-2025, and Google’s developer advocacy hasn’t pushed it hard. Most Gemini users know about Google Search grounding and code execution, but URL context flies under the radar.
Part of the problem is naming: “URL context” sounds like a basic citation feature, not a full-fledged web scraper. If Google had called it “Live Web Data” or “URL Scraper,” adoption would be higher.
The other issue: it’s not available via traditional function calling (the method most devs use for tool integration). You have to use the new tools API, which requires updating your code structure. That friction slows adoption.
The Bottom Line
If you’re using the Gemini API and you’re not leveraging URL context, you’re leaving value on the table. It’s free (tokenized), multimodal, and fast. For most scraping use cases – documentation, PDFs, competitive research – it beats dedicated services on simplicity and cost.
The limits (20 URLs per request, 34MB per URL, public URLs only) matter for high-scale scraping, but for agentic workflows, RAG pipelines, and ad hoc research, URL context is a Tier 1 tool.
Download the Gemini SDK, enable URL context in your tools config, and test it against your current scraping stack. I’d bet 80% of your use cases can shift to URL context with zero external dependencies.
Google built one of the best web scrapers in the Gemini API, and almost nobody knows it exists. Now you do.
FAQ
Is Gemini URL context really free?
Yes, in the sense that there’s no per-scrape fee. You pay only for the input tokens resulting from the scraped content, at Gemini’s standard token pricing (~$0.125 per million input tokens for Gemini 3 Flash).
Can Gemini URL context scrape JavaScript-rendered content?
It depends. If the content is in Google’s indexed cache and was rendered during indexing, yes. For live fetches, results vary based on how the page is structured. For complex JS-heavy sites, dedicated headless browser scrapers may still be better.
How does URL context handle rate limiting?
Gemini manages this internally. If a site blocks Google’s scraper, the request will fail, but you’re not managing retry logic or proxy rotation yourself. For high-frequency scraping, dedicated services with built-in rate limit handling are more robust.
