Google’s vast database contains information from over 130 trillion individual pages and, when a user types something into the search engine, a search algorithm parses through this enormous wealth of knowledge to deliver results it thinks are relevant to that particular search.
This database is referred to as an ‘index’. You want your website and its pages to be included in this index, simply because, once they are, you can start jostling for the top spot in the all-important search results. This will help you establish yourself in your particular sector, attract traffic and, hopefully, convert visitors into customers.
Crawling vs indexing
Google indexes your site through a method called ‘crawling’ – this is where bots or spiders trawl through the pages on your website and rank them in the index depending on how useful, relevant and valuable they are to search engine users. These spiders are always searching for new, updated content, but, if your site has lots of new pages, they could take time to find, meaning there is a delay in terms of how visible you are in the search results. You need to ensure your site is indexed as soon as possible – according to Neil Patel, only 5.7% of new pages make it on to the first page of the SERPs within a year of publication.
Is my website already indexed?
Unless your website is very new, it’s likely that it will be indexed. However, there’s still probably plenty of room for improvement. In this blog, we’ll take you a number of ways in which you can improve your website’s indexation.
4 indexation best practices
1. Install Google Analytics & Search Console
You may already have these tools installed, but if not, it’s vital that you add them to your repertoire. If you don’t, you’re effectively blind to how Google indexes your site. Google Analytics records visits to your website and gives you insight into visitor habits, such as how long they spend on your site and which pages they tend to look at. Google Search Console, which was previously known as Google Webmaster Tools, lets you see when your website was last crawled. It also alerts you to any issues with indexing, or security problems. For help with getting these set up, you can consult LunaMetrics’ helpful guide.
2. Create and submit a sitemap
Create a sitemap using a free plugin such as Google XML Sitemaps and submit it to Google Search Console to tell the search engine how often it should crawl your site. Now that you’ve created a sitemap, you’ll need to submit it to Search Console. After you’ve done this, you can also use the Index Coverage Status report in Search Console to find out which pages have been indexed and how you can fix those that haven’t.
3. Be proactive
There are a number of SEO best practices you can use to get Google to index your site faster. These include on-site improvements, such as creating links pointing to your new pages and regularly blogging, as well as off-site improvements: earning inbound links and sharing on social media. You can find out about more indexation best practices here, or consult SEO professionals.
4. Mobile-first indexing
Mobile-first indexing has been rolled out by Google this year and reflects the increasing likelihood that visitors will be using their smartphone – rather than their desktop – to access your site. ‘Mobile-first’ refers to Google considering the mobile version as the primary version of your site. If you don’t have a mobile-friendly site, the desktop experience can still be included in the index, but it’s likely that you’ll fall behind competitors who do cater for those using mobiles.
Although it’s still in its early stages, ‘mobile-first’ represents a shift in the way Google treats your site. If your site is responsive anyway, it’s likely that you won’t have to change anything for now, but you should ensure that visitors’ mobile experience is optimised, with quick page load times and easy navigation. There should also be consistency between the mobile and desktop versions of your site.
Common mistakes that lead to no-index
If some of your pages aren’t being indexed by Google, there’ll usually be a reason for it. Here are a few common mistakes that can lead to no index:
Pages not loading properly
URLs can accidentally get redirected and produce 404 or 500 errors, meaning they’re unlikely to be indexed. Ensure your URLs are getting a 200 status response code.
Submitted URL blocked by ‘robots.txt’
Your website’s robots.txt file gives Google its crawl commands. If a page is missing from Google’s index, check here. If it says: google-index-blocked-robots-txt, it will have previously indexed a page on your site that is now blocked via robots.txt.
Large amounts of internal or external duplicate content could be a reason for Google not indexing your page. Use Siteliner to crawl your website: this will highlight any duplicate content, making it easier for you to rectify this particular problem. Copyscape can help deal with any matches to other sites.
Pages without internal links are referred to as ‘orphaned pages’. Google’s spiders have no way of knowing these pages exist, so won’t index them.
Check for Google crawl errors frequently
It’s vital that you check for Google crawl errors, such as 404s, preferably at least once a month. To do this, open up Search Console and select ‘Crawl Errors’ from the ‘Crawl’ menu. If any errors appear, you should take measures to rectify these as soon as possible – otherwise, certain pages may not be being indexed, you could develop a reputation for providing a poor user experience and you could see your ranking fall as a result.
Summary of indexation best practices
To summarise, when it comes to indexation, you should install tools such as Analytics and Search Console, create and submit a sitemap and be proactive in terms of linking back to your new pages. You should also ensure you comply with mobile-first indexing best practices and regularly check for Google crawl errors.