Crawl examples
Docs subtree crawl
Crawl a docs portal
```bash
curl https://api.faircompany.ai/v1/crawl/web/crawl \
-X POST \
-H "Authorization: Bearer fc_live_xxx" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.stripe.com",
"limit": 20,
"render": false
}'
```
```ts
const crawl = await fc.scrape.crawl({
url: "https://docs.stripe.com",
limit: 20,
render: false,
});
console.log(crawl.pagesScraped);
console.log(crawl.pages[0]?.url);
```
```python
crawl = fc.crawl(
"https://docs.stripe.com",
limit=20,
render=False,
)
print(crawl.pages_scraped)
print(crawl.pages[0].url if crawl.pages else None)
```
```json
{
"tool": "faircrawl_crawl",
"arguments": {
"url": "https://docs.stripe.com",
"limit": 20,
"render": false
}
}
```
Regex-constrained crawl
curl https://api.faircompany.ai/v1/crawl/web/crawl \ -X POST \ -H "Authorization: Bearer $FAIRCRAWL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://docs.stripe.com", "limit": 30, "regexOnFullURL": "^https://docs\\.stripe\\.com/payments/.*" }'Render-aware crawl
If the site exposes content only after client-side rendering, set render: true and keep the limit low while you validate the output.