Understand Website Crawlers

Website Crawlers read public HTML pages from an XML sitemap and make their main content searchable by linked assistants.

The crawler list shows the sources available to you and their current status. Open a crawler to manage its schedule, start or stop work, and inspect each page.

Only public HTML pages are indexed. Files such as PDFs, images, videos, and downloads are skipped. Use a Knowledge Base for supported uploaded documents.

After starting a crawl, you can leave the page and return later to review its progress and notifications.