> ## Documentation Index
> Fetch the complete documentation index at: https://docs.truscan.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Discover the URLs on a site



## OpenAPI

````yaml /api-reference/crawl.json post /api/crawl/map
openapi: 3.1.0
info:
  title: Truscan Crawl
  version: 1.0.0
  description: >-
    Walking a website: what is on it, and all of it.


    `/api/crawl/map` answers the first question and returns synchronously,
    because discovery is a handful of fetches rather than hundreds: a site that
    publishes an accurate sitemap costs one request to map.


    `/api/crawl` answers the second and returns a job, because a crawl of two
    hundred pages takes minutes and no HTTP client should hold a connection open
    for that long. Poll the job by id.


    Both respect `robots.txt`, including its crawl delay. This is not politeness
    for its own sake: every request the platform makes leaves from one address,
    so a crawler that ignores a site's wishes does not get that site blocked, it
    gets search and extract blocked for everybody.


    Results are kept for seven days and then deleted. That window is a TTL on
    the store rather than a policy document, so nothing outlives it by being
    forgotten about.
servers:
  - url: https://api.truscan.co
security:
  - bearerAuth: []
paths:
  /api/crawl/map:
    post:
      summary: Discover the URLs on a site
      operationId: mapSite
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/MapRequest'
      responses:
        '200':
          description: The URLs found, and where they came from
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/MapEnvelope'
        '401':
          $ref: '#/components/responses/Failure'
        '403':
          description: The site's robots.txt asks crawlers not to read that address.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Failure'
        '422':
          $ref: '#/components/responses/Failure'
        '429':
          $ref: '#/components/responses/Failure'
components:
  schemas:
    MapRequest:
      type: object
      required:
        - url
      properties:
        url:
          type: string
          description: >-
            The site to map. A bare hostname is accepted and assumed to be
            https.
          examples:
            - https://example.com
        limit:
          type: integer
          minimum: 1
          maximum: 5000
          default: 5000
          description: How many URLs to return.
        search:
          type: string
          description: >-
            Keep only URLs containing this text, matched case-insensitively
            against the whole URL. This is what makes a map of a ten thousand
            page site useful rather than merely large.
          examples:
            - docs
        subdomains:
          type: boolean
          default: false
          description: >-
            Widen the site from one host to its subdomains, so mapping
            example.com also lists docs.example.com.
        sitemap_only:
          type: boolean
          default: false
          description: >-
            Read the sitemap and stop, skipping link discovery. Faster, kinder
            to the site, and complete for anyone who publishes an accurate
            sitemap.
    MapEnvelope:
      type: object
      properties:
        success:
          type: boolean
          const: true
        message:
          type: string
        result:
          $ref: '#/components/schemas/MapResult'
    Failure:
      type: object
      properties:
        success:
          type: boolean
          const: false
        message:
          type: string
        result:
          type: object
          properties:
            code:
              type: string
    MapResult:
      type: object
      properties:
        url:
          type: string
        urls:
          type: array
          items:
            type: string
        total:
          type: integer
        sources:
          type: object
          description: >-
            How many URLs each method contributed, so you can tell an accurate
            sitemap from a site we had to walk.
          properties:
            sitemap:
              type: integer
            links:
              type: integer
        truncated:
          type: boolean
          description: The limit cut the list short, rather than the site being that size.
  responses:
    Failure:
      description: >-
        The envelope with success false. result carries the machine-readable
        code and nothing else; the HTTP status line carries the real status.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Failure'
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      description: >-
        An API key, sent as `Authorization: Bearer enc_…`. A dashboard session
        cookie works on the same route.

````