Review and control crawled pages

Understand page status

• Pending is waiting for processing.
• Processing is being fetched and indexed.
• Ready is available to linked assistants.
• Cancelled was stopped before completion.
• Error failed and needs an explicit retry.
• Excluded is deliberately blocked from search and later sitemap runs.

Use status filters and pagination to find the pages you need.

Exclude or restore a URL

Select Exclude to remove a page from active search and block it from later sitemap runs. Its existing searchable content is also removed.

Open the Excluded filter and select Restore to allow the URL again. It can return on a later crawl if it remains in the sitemap.

Review page history

Select History to see up to the most recent 50 attempts, including indexed, unchanged, failed, in-progress, cancelled, denied, or rejected outcomes.

Inspect metadata

Use Metadata to review the canonical reference ID, extraction source, indexed generation, chunk count, and structured catalog facets.

Explicit Doublewire page markers are preferred. Schema.org productID and SKU are used only when the crawler’s fallback setting allows them. A page without a reference can still answer normal knowledge questions, but cannot provide a canonical item reference for later actions.