Rate limits

FairCrawl applies a Redis-backed 60-second counter window per org for the main crawl routes.

Current defaults

Action	Limit per 60 seconds
scrape	100
research	20
map	50
interact	10
agent	5

What happens on limit hit

• the API returns 429
• Retry-After is included in seconds
• the current request is rejected before any billable work is dispatched

Backoff example

until curl https://api.faircompany.ai/v1/crawl/web/scrape \
  -X POST \
  -H "Authorization: Bearer $FAIRCRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'; do
  sleep 5
done

for (let attempt = 0; attempt < 5; attempt += 1) {
  try {
    return await fc.scrape({ url: "https://example.com" });
  } catch (error) {
    const retryAfter = Number((error as any)?.details?.retryAfter || 0);
    await new Promise((resolve) => setTimeout(resolve, Math.max(retryAfter, 1) * 1000));
  }
}

import time
from faircrawl import FairCrawl
from faircrawl.errors import FairCrawlError

fc = FairCrawl()

for _ in range(5):
    try:
        result = fc.scrape("https://example.com")
        break
    except FairCrawlError:
        time.sleep(5)

Notes

• one-off org overrides exist in code for exceptional cases
• rate limiting is per org, not per API key
• read-only docs pages and the public product website are unaffected