Web Scraping API
Scrape any website with CSS selectors, XPath, or AI — with built-in proxy rotation, JS rendering, and smart retries.
The Agenty Scrape API is a production-grade scraping endpoint for static and JavaScript-rendered sites. Use CSS selectors, XPath, or AI-powered schema detection to pull the data you need, while the API handles rotating proxies, rate limits, retries, and bot-detection countermeasures for you.
Features
- CSS & XPathBoth selector syntaxes with attribute pickers.
- AI schema detectionDescribe fields in plain English; AI finds them.
- JS renderingHeadless Chrome for SPAs and dynamic pages.
- Auto paginationFollow next links or infinite scroll.
- Rotating proxiesResidential and datacenter, geo-targeted.
- SessionsMaintain cookies across multi-step flows.
- Smart retriesBackoff on transient errors and bot blocks.
- Batch modeSubmit thousands of URLs in one job.
Use cases
- E-commerce price, stock, and review monitoring
- Real-estate and rental listing aggregation
- Job board aggregation and salary benchmarking
- Travel rate and availability comparison
- Public social-profile and engagement analytics
API examples
curl -X POST https://api.agenty.ai/v1/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/products",
"selectors": {
"products": {
"_parent": ".product-card",
"name": ".title",
"price": ".price",
"image": "img@src"
}
}
}'curl -X POST https://api.agenty.ai/v1/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/products",
"selectors": {
"names": "//h2[@class=\"title\"]/text()",
"prices": "//span[@class=\"price\"]/text()"
}
}'const res = await fetch('https://api.agenty.ai/v1/scrape', {
method: 'POST',
headers: {
'Authorization': 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json',
},
body: JSON.stringify({
url: 'https://example.com/products',
selectors: { name: 'h1', price: '.price' },
}),
});
console.log(await res.json());import requests
res = requests.post(
"https://api.agenty.ai/v1/scrape",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"url": "https://example.com/products",
"selectors": {"name": "h1", "price": ".price"},
},
)
print(res.json())How Agenty compares
| Feature | Agenty | ScrapingBee | ScraperAPI | ZenRows |
|---|---|---|---|---|
| CSS selector support | Yes | Yes | Yes | Yes |
| XPath support | Yes | Yes | Limited | Yes |
| AI schema detection | Yes | No | No | No |
| JavaScript rendering | Yes | Yes | Yes | Yes |
| Free tier | Yes | Yes | Yes | Yes |
Frequently asked questions
What is the Web Scraping API?
The Agenty Scrape API lets you extract data from any website using CSS selectors, XPath, or AI-powered schema detection. It handles JavaScript rendering, rotating proxies, pagination, and retries automatically.
Do I need to manage proxies?
No. We manage a global rotating proxy pool. Optionally pass proxyCountry to route requests through a specific region.
Can I scrape single-page apps (SPAs)?
Yes. We use headless Chrome to render JavaScript before extraction. React, Vue, and Angular single-page apps are fully supported.
How do you handle bot detection?
We use rotating residential proxies, realistic browser headers, and randomised delays to minimise detection. For tougher targets, we offer CAPTCHA-solving add-ons.
Is there a free tier?
Yes. All accounts start with free credits. Visit our pricing page for current limits.