Skip to content
FairCrawl Docs

Rate limits and errors

FairCrawl returns simple HTTP problem details. The practical production split is:

  • 400 for validation or malformed requests
  • 401 / 403 for auth failures
  • 402 for billing problems
  • 429 for org-scoped rate limits
  • 5xx for upstream or worker-side failures

A typical 429-safe shell loop

Terminal window
until curl https://api.faircompany.ai/v1/crawl/web/scrape \
  -X POST \
  -H "Authorization: Bearer $FAIRCRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'; do
  sleep 5
done

TypeScript error handling

import { FairCrawl, FairCrawlError } from "faircrawl";


const fc = new FairCrawl({ apiKey: process.env.FAIRCRAWL_API_KEY! });


try {
  await fc.scrape({ url: "https://example.com", proxy_country: "usa" });
} catch (error) {
  if (error instanceof FairCrawlError) {
    console.error(error.status, error.code, error.detail);
    console.error(error.requestId, error.details);
  }
}

Python error handling

from faircrawl import FairCrawl
from faircrawl.errors import FairCrawlError, RateLimitError


fc = FairCrawl()


try:
    fc.scrape("https://example.com", proxy_country="usa")
except RateLimitError as error:
    print("retry after", error.retry_after)
except FairCrawlError as error:
    print(error.status, error.code, error.message)

Notes

  • Retry-After is present on current rate-limit responses
  • the TypeScript and Python SDKs normalize request timeout and network failures into typed errors
  • validation errors often include a narrow field-level message in detail