Creating content is time-consuming and costly; website owners want search engines to index content changes as quickly as possible. This time lag results in missed opportunities for businesses to engage with their audience or even attract prospective new customers. Yandex and Microsoft bring a new protocol, IndexNow, enabling webmasters to instantly index their content by the search engine as soon as they click the publish button. This protocol uses the Application Programming Interface (API) to ping search engines and immediately notify them about the changes you make on your page.
Table of Contents
What is indexing?
Content indexing is the process where search engines identify new content that you publish online. The search engines use this to determine the function of your website and the pages on it. This indexing is done through a process known as crawling, where a web spider or web robot carries out an automated fetch of web pages and analyses the content of a page. Search Marketers have to ensure that their content is well-positioned for indexation. The search engine finds your website, adds it to its index, relates each page with searched topics, returns the site on Search engine result pages (SERPs), and drives the right audience to see your content. If your content isn’t indexed, there will be no organic traffic, leads, or conversion.
The IndexNow protocol seeks to eliminate the process of depending on a search engine crawler to find each URL to index. Through this, websites notify the search engines participating in this initiative of their new, updated, or deleted content. This information enables the search engines to prioritize crawl for these URLs, which eliminates the process of exploratory crawls, which takes time to identify content changes. This protocol is open source in that any search engine is free to participate. All the participating search engines get notified when one search engine is notified of an update or change.
Types of crawls
Two main types of crawls get content from the website.
- Page crawls, where a crawler seeks to crawl a single page or a blog post
- Site crawls, where the spider crawls the entire site at one time. Starting with the homepage, it will grasp links from the page and crawl the site to other content.
How it works
The IndexNow is easy to adopt, and here are the simple steps to follow in adopting this protocol.
- Generate a key supported by the protocol. You can achieve this by using an online key generator tool to generate an Indexing API key. This key is required to match the submitted URLs with the ownership of the domain. It is recommended to place the key on your site’s root directory and create a separate key for each subdomain on the site.
- Host the key in a text file named with the value of the key at the root of your website, which helps verify the ownership of the URLs submitted. The GET method is recommended when sending a single address and if sending many addresses, use the POST method.
- Submit the URLs when they are added, updated, or deleted. Web admins can submit a single URL or a bulk of them as per the API call.
Your key and a simple HTTP request containing the changed URL is the only thing you need to work through the IndexNow protocol.
Who can use this protocol?
Developers can use this protocol through the use of the steps described above. This indexing protocol is not limited to developers only. If you are a non-developer, there is no cause to worry since popular platforms are adopting this protocol, and if you are using one of these platforms, you will have cared. Large websites such as eBay, GitHub, and LinkedIn are planning to migrate to IndexNow. Web content management systems (CMS) are also a target for this protocol. IndexNow is encouraging them to adopt this so that their users can benefit from getting their latest content indexed immediately. Microsoft provided a code for WordPress inclusion into the core without requiring a plugin. Wix and Duda are planning to integrate with this protocol soon. SEO companies such as Botify and Oncrawl are also adopting this initiative. Content Delivery Networks (CDN) such as Akamai can also integrate into this protocol with the Global network platform Cloudflare supporting this initiative.
Impact of CDNs on web indexing
CDNs help speed up data delivery by bringing it closer to the users. Using proxy servers, they cache web content like images, videos, and web pages near your physical location. This cache enables you to do things like posting on social media, downloading software, checking your bank balance, and making purchases without waiting for content to load. With the excellent view that the CDNs have of the web, they are perfect for tracking bot and human traffic. Cloudflare has an incredible recognition of when URLs change, thus helping search engines identify changes efficiently. Cloudflare is crucial to this initiative and provides more efficient web indexing.
The Bing URL submission API
Both the Bing URL submission API and the IndexNow have a similar aim of reducing crawling and improving internet efficiency. The Bing URL submission API allows websites to submit their URLs and content changes immediately. Still, despite being similar to the IndexNow, the changes are only received by the Bing search engine, and the IndexNow allows other search engines participating to receive the changes.
Despite being asked to participate, Google is not yet an active participant in this initiative.
IndexNow and XML sitemaps
XML sitemaps are a list of URLs available to a site. A sitemap helps search engines find, crawl and index your website’s content. IndexNow informs the search engines about new URLs, and they don’t have to crawl XML sitemaps. Search engines tend to prioritize URLs submitted through this protocol over those found in other ways.
Non-indexed pages
Submitting your URLs does not guarantee that all will be indexed, and these are referred to as non-indexed pages. Here are some of the reasons why your page might not get indexed.
- Page redirecting to 404, server errors, or page not found.
- Duplicate content on pages.
- Poor structure.
- Lack of enough authority to justify all the web pages.
Importance of indexing
Traffic is the backbone of the success of your online business. Without having your content indexed by the search engines, you will not get any traffic. It helps you have healthy search visibility. When you allow the search engine to access a lot of information about your site after indexing, this will give you a search advantage and improve your visibility.
Advantages of using this initiative
- The prime advantage of using this protocol is reducing the time for content to be discovered and indexed. By eliminating the dependence on spiders, site owners submit their website content for indexing, giving them crawl priority. The process takes lesser time than it takes with organic indexing.
- With the other indexing protocols, you might not get the spiders to crawl your site. This protocol gives you the advantage of having your pages crawled, thus having a high chance of instant indexing. Without this, It can take days to weeks for search engines to discover website changes or updates since not all URLs are crawled regularly with organic crawling.
- It utilizes the push search indexing method, and this gives you, the site owner, the advantage of informing the search engines when you have published or updated. This process is contrary to the pull method which search engines use, which requires the crawler to visit the site and pull data from the server.
- With Cloudfare’s crawler hints supporting this protocol, with the click of a button, it will take the responsibility of informing the search engines when your content has changed.
- This protocol helps reduce the server load by lowering crawling demands; this allows the server to perform optimally without the burden of redundant web pages that search engines already have.
- Improved relationship between the publishers and search engines, through indexing efficiency, is a prime benefit to both.
- This protocol has a greener effect on the energy we use. Reducing the crawling demands made by the conventional indexing method and eliminating the crawling of entire web pages eliminates the relentless electricity use.
This protocol is the ultimate future of web indexing. Having many search engines joining Microsoft Bing Index Now and Yandex in participating in this protocol, a better internet experience is guaranteed for both the web users and the search engines. This protocol has a revolutionary effect in the world of search by providing the users with the relevant and freshest results in outdated content, eliminating the use of exploratory crawls, thus improving efficiency and giving the website owners the benefit of having their quality content paying off faster. It is important to note that having your content indexed faster does not necessarily mean ranking you better in search results. You have to create high-quality, unique, and SEO-optimized content.
You can learn more about implementing this process at indexnow.org.
FAQ
What is IndexNow?
