Knowledge that finds itself
Pointing an agent at your site used to mean hoping you had a sitemap, and hoping your URLs happened to contain the words people search for. Both assumptions are gone.
Pages are indexed by what they say, not what their URL says
Discovered pages now carry their title and meta description, so "how long do I have to send something back?" matches a page called "Returns & Refunds" instead of having to match the string "policies returns". Pages at URLs like /p/48213 were previously invisible to search; they are not any more.
Read moreLink crawling for sites without a sitemap
No sitemap? We follow links from the page you gave us, staying on the same origin, with page and depth caps. robots.txt is respected throughout — Disallow rules are honoured, and a Crawl-delay makes the crawl serial at the interval the site asked for.
Read moreInclude and exclude path rules
Index /docs and skip /cart. On a real storefront this is the difference between an agent whose knowledge is ten thousand near-identical product pages and one that can actually find your twelve policy pages.
Read moreChoose live pages or a stored index, per source
Keep the default and pages are fetched at answer time — no storage cost, and edits show up immediately. Turn on full content indexing for a docs site where precision matters more than breadth. Mix both across sources.
Read more