Crawling, indexing, ranking
Search engines work in three steps. They crawl, following links and sitemaps to fetch pages. They index, reading each page, deciding whether it is worth keeping and which URL is the canonical one. Then they rank the indexed pages for each query. A page can be crawled and still not be indexed: Google's own reports call this "Crawled, currently not indexed".
Why a page is not indexed
The common reasons are a noindex tag or header, a canonical tag pointing to another URL, a robots.txt rule that blocks crawling, a redirect, an error status code, or content that is thin or nearly the same as another page. On a new site the most common reason is simply time: Google has not decided the site is worth crawling deeply yet.
Search Console's URL Inspection tool shows the status of any single URL, and the Pages report shows every reason pages were left out.
How to get pages indexed faster
Submit an XML sitemap in Search Console and keep it to pages you want indexed. Link new pages from pages that are already crawled often, such as your homepage. Use IndexNow to tell Bing and other engines that support it about new and changed URLs. Earn a few links from other sites: a new domain with no links is crawled slowly. And make each page worth indexing, with content that is not on any other page of your site.
Example
A startup publishes 40 comparison pages. Two weeks later Search Console shows 12 indexed and 28 as "Discovered, currently not indexed". The founder links the remaining pages from the homepage and a hub page, and most are indexed within the following month.
Questions people ask
- How do I check if a page is indexed?
- Inspect the URL in Google Search Console. A search for site:yourdomain.com/page gives a rough answer but is not reliable.
- How long does indexing take?
- From a few hours on an established site to several weeks on a new domain. Links from crawled pages and a clean sitemap shorten it.
Related terms
- NoindexNoindex is a directive that asks search engines to keep a page out of their results. It is set with <meta name="robots" content="noindex"> or an X-Robots-Tag HTTP header.
- XML sitemapAn XML sitemap is a file that lists the URLs you want search engines to crawl and index, optionally with the date each page last changed.
- Crawl budgetCrawl budget is the number of URLs a search engine is willing and able to fetch from your site in a given time. It matters most for large sites and sites with many low value URLs.
- Canonical URLA canonical URL is the one address you want search engines to index for a page that can be reached at several. You declare it with <link rel="canonical" href="..."> in the page's head.
- IndexNowIndexNow is an open protocol that lets a website notify search engines the moment a URL is added, updated or deleted, instead of waiting for them to crawl it.
- robots.txtrobots.txt is a plain text file at the root of a site that tells crawlers which paths they may and may not fetch. It controls crawling, not indexing.