> ## Documentation Index
> Fetch the complete documentation index at: https://docs.truscan.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Start crawling a site

> Returns as soon as the job exists. The crawl runs afterwards; read it back by id.



## OpenAPI

````yaml /api-reference/crawl.json post /api/crawl
openapi: 3.1.0
info:
  title: Truscan Crawl
  version: 1.0.0
  description: >-
    Walking a website: what is on it, and all of it.


    `/api/crawl/map` answers the first question and returns synchronously,
    because discovery is a handful of fetches rather than hundreds: a site that
    publishes an accurate sitemap costs one request to map.


    `/api/crawl` answers the second and returns a job, because a crawl of two
    hundred pages takes minutes and no HTTP client should hold a connection open
    for that long. Poll the job by id.


    Both respect `robots.txt`, including its crawl delay. This is not politeness
    for its own sake: every request the platform makes leaves from one address,
    so a crawler that ignores a site's wishes does not get that site blocked, it
    gets search and extract blocked for everybody.


    Results are kept for seven days and then deleted. That window is a TTL on
    the store rather than a policy document, so nothing outlives it by being
    forgotten about.
servers:
  - url: https://api.truscan.co
security:
  - bearerAuth: []
paths:
  /api/crawl:
    post:
      summary: Start crawling a site
      description: >-
        Returns as soon as the job exists. The crawl runs afterwards; read it
        back by id.
      operationId: startCrawl
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CrawlRequest'
      responses:
        '202':
          description: The crawl was accepted and has not been done yet.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/JobEnvelope'
        '401':
          $ref: '#/components/responses/Failure'
        '422':
          $ref: '#/components/responses/Failure'
        '429':
          description: >-
            Either the per-IP rate limit, or as many crawls already running at
            once as we allow.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Failure'
components:
  schemas:
    CrawlRequest:
      type: object
      required:
        - url
      properties:
        url:
          type: string
          description: Where to start. A bare hostname is accepted and assumed to be https.
          examples:
            - https://example.com
        max_pages:
          type: integer
          minimum: 1
          maximum: 500
          default: 50
          description: >-
            How many pages to read. Discovery may find many more, and the job
            reports both numbers so you can tell a budget that ran out from a
            site that ended.
        max_depth:
          type: integer
          minimum: 1
          maximum: 5
          default: 5
          description: >-
            How far from the start to follow links. Depth is multiplicative, so
            raising it costs far more than raising max_pages.
        include:
          type: array
          items:
            type: string
          description: >-
            Path patterns to follow. Empty means everything not excluded. `*`
            matches anything but a slash, `**` anything, `?` one character.
          examples:
            - - /docs/**
        exclude:
          type: array
          items:
            type: string
          description: >-
            Path patterns to skip. An exclude always beats an include, so "all
            of /docs except /docs/internal" says exactly that.
          examples:
            - - /docs/internal/**
        subdomains:
          type: boolean
          default: false
          description: Follow links onto subdomains of the starting host.
        max_chars:
          type: integer
          description: Cap the text kept per page. Zero uses the service default.
        structured:
          type: boolean
          default: false
          description: >-
            Include each page's own machine-readable declarations: JSON-LD,
            OpenGraph, Twitter cards, microdata, feeds and contacts.
        tables:
          type: boolean
          default: false
          description: Include the tables recovered from each page, as rows of cells.
    JobEnvelope:
      type: object
      properties:
        success:
          type: boolean
          const: true
        message:
          type: string
        result:
          $ref: '#/components/schemas/Job'
    Failure:
      type: object
      properties:
        success:
          type: boolean
          const: false
        message:
          type: string
        result:
          type: object
          properties:
            code:
              type: string
    Job:
      type: object
      properties:
        id:
          type: string
          examples:
            - crwl_8d3ddecc038c10d4
        status:
          type: string
          enum:
            - queued
            - running
            - completed
            - failed
            - cancelled
        url:
          type: string
        discovered:
          type: integer
          description: >-
            URLs the crawl found. Exceeds `completed` when the page budget
            stopped it reading them all, which is the number that tells you to
            ask for more.
        completed:
          type: integer
          description: Pages read successfully. This is what is billed.
        failed:
          type: integer
          description: Pages that could not be read. Not billed.
        max_pages:
          type: integer
        created_at:
          type: string
          format: date-time
        updated_at:
          type: string
          format: date-time
        finished_at:
          type: string
          format: date-time
        expires_at:
          type: string
          format: date-time
          description: >-
            When this crawl and its pages are deleted. Reset on every write, so
            a long crawl cannot expire while it is still running.
        error:
          type: string
          description: Why a failed crawl stopped, in a sentence you can act on.
  responses:
    Failure:
      description: >-
        The envelope with success false. result carries the machine-readable
        code and nothing else; the HTTP status line carries the real status.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Failure'
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      description: >-
        An API key, sent as `Authorization: Bearer enc_…`. A dashboard session
        cookie works on the same route.

````