Crawl parameters
| Field | Type | Required | Notes |
|---|---|---|---|
url | string | yes | Seed URL. |
limit | number | no | Maximum number of pages to scrape. |
render | boolean | no | Render JS while crawling. |
regexOnFullURL | string | no | Filter URLs using a full-URL regex. |
deduplicateSimilarURLs | boolean | no | Merge near-duplicate URLs before scrape work. |
callback_url | string | no | Callback on completion for async workflows. |
Guidance
- keep
limitlow until you know the target site structure - use
regexOnFullURLon docs or changelog sites to stay on the exact subtree you need - enable de-duplication when the site emits tracking-heavy URL variants
Crawl parameter baseline
```bash
curl https://api.faircompany.ai/v1/crawl/web/crawl \
-X POST \
-H "Authorization: Bearer fc_live_xxx" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.stripe.com",
"limit": 20,
"render": false
}'
```
```ts
const crawl = await fc.scrape.crawl({
url: "https://docs.stripe.com",
limit: 20,
render: false,
});
console.log(crawl.pagesScraped);
console.log(crawl.pages[0]?.url);
```
```python
crawl = fc.crawl(
"https://docs.stripe.com",
limit=20,
render=False,
)
print(crawl.pages_scraped)
print(crawl.pages[0].url if crawl.pages else None)
```
```json
{
"tool": "faircrawl_crawl",
"arguments": {
"url": "https://docs.stripe.com",
"limit": 20,
"render": false
}
}
```