Skip to content

Crawl response shape

{
"seed": "https://docs.stripe.com",
"pages": [
{
"url": "https://docs.stripe.com/payments",
"title": "Payments",
"description": "Stripe payments docs",
"text": "Extracted page text...",
"statusCode": 200
}
],
"links": [
{ "url": "https://docs.stripe.com/checkout", "text": "Checkout" }
],
"totalLinks": 181,
"pagesScraped": 20,
"limit": 20,
"_meta": {
"requestId": "req_crawl123",
"platform": "web",
"action": "crawl",
"cost": 0.014,
"responseTimeMs": 3148
}
}

Interpretation

  • pages contains the actual fetched documents
  • links is the discovered edge list FairCrawl saw while expanding
  • pagesScraped is the final bounded count
  • _meta.cost reflects the actual scrape work underneath the crawl

Reproduce the crawl payload

```bash curl https://api.faircompany.ai/v1/crawl/web/crawl \ -X POST \ -H "Authorization: Bearer fc_live_xxx" \ -H "Content-Type: application/json" \ -d '{ "url": "https://docs.stripe.com", "limit": 20, "render": false }' ```