What is a Sitemap?
A sitemap is a way to organize a website while identifying the URLs and data under each section. The XML document contains instructions for search engine bots.
Also, a sitemap can be defined as a diagrammatic representation that shows how subpages are arranged beneath the parent groupings. This kind of diagram forms a map of a site. This map is an XML file which enables search engine bots to index a site. Sitemaps can also be defined as a demonstration of the navigation layout and how content is organized on a website.
The Meaning Behind Sitemaps
Websites continue to increase in size and complexity. For instance, a site like Amazon probably has thousands of pages and maybe millions of products. How easy would it be for search engine bots to crawl such a website? How does SEO really work?
Perhaps Amazon is at an extreme example. Let’s look at a business website instead. It has the homepage that gives an overview of the business, the product or services it offers, contact information, blog, and maybe information about the team. Each of these components is further explained on separate pages and each page probably has smaller subsections.
For a blog, there could be anchor text that links to other sections of the site in addition to the content that is already frequently updated—sometimes several times a day for news sites. What about when you have links that point to other websites or web pages? How about discussion forums in which a particular topic can get hundreds of comments and links to other sites?
Needless to say, websites can get quite complex. Search engines were getting to a point of being unable to track the complex structures. Some pages, especially dynamic pages, kept being overlooked.
Unlike popular belief, sitemaps are not a search engine optimization tool; they also do not affect ranking. Instead, they are aimed at helping search engines make accurate rankings and searches.
Google developed a sitemap protocol in 2005 known as the Sitemap 0.84 Protocol to solve this problem. It is an XML protocol which was published under a Creative Commons License. This allowed other search engines to support it. Website developers, therefore, do not need to create different sitemap types. All they need to do is create a single file for submission and update it when changes are made to the site.
Why are Sitemaps Important?
Though the answer to this question may be clear from the above discussion of sitemaps, here are a few specific reasons sitemaps are important:
1. Help Search Engine Bots Index All the Pages on a Website.
Search engine bots previously overlooked dynamic pages. The XML protocol, however, allowed the bots to track URLs more efficiently. This is because the necessary information is placed on a single page. Since the XML also contains a summary of how frequently a website is updated and keeps a record of when changes are made, it becomes easier for the bots to crawl web pages and URLs. The data that the search engines need is all accessible in one location, thus less time is spent on a website (remember, bots have millions of sites to crawl).
Sitemaps allow search engine bots to crawl even URLs that are not discovered through the normal crawling process.
2. Easier Navigation
We mentioned earlier that sitemaps were initially meant for users. When a sitemap is designed for users, it is usually a list of web pages or even web design planning tools with a hierarchical arrangement, appearing in HTML format. The sitemap can list all the pages on your site. Any updates to your site need to be added to the sitemap (for example, new images, videos, pages, and any other content).
When you have a sitemap on your site, it is easier for a visitor to find a particular page or topic. This is because you organize your content via topic or category. The sitemap helps you organize your content in a way that the core content is near the homepage for easy accessibility. The sitemap contains anchor text that points to the most important parts of your site. Visitors can use the sitemap to locate a topic by navigating through the menus or searching the site.
An HTML sitemap is also useful in letting your visitors know what your website is all about. If you have a number of links on your sitemap, you may want to split them into multiple pages. When visitors can easily find what they are looking for, they spend more time on your site, engage in more sessions and maybe even visit a lot of pages per session if you have good content. You will also ultimately have repeat visitors and conversions.
You may wonder why you need a sitemap instead of concentrating on building a good UX for your website visitors. The truth is it is difficult to achieve 100% UX. From time to time, users will begin to look for an internal site or re-type a query, or even look for navigation elements to try and locate a particular page or topic—this is where sitemaps come in.
3. Easier Web Development
Just like we have previously discussed, Google developed the Sitemap 0.84 Protocol under a Creative Commons License. This was to encourage other search engines to adopt the protocol, as Yahoo and Microsoft agreed to do shortly after. As a result, the sitemap was updated to version 0.9 and the three competitors have jointly sponsored a site to explain the protocol in detail. See here.
What does this mean for developers? It means that the protocol is recognized across main search engines, thus web developers need not create different types of sitemaps for each search engine. All they need to do is create a single file and submit it, then update it as soon as any changes are made on a site.
Over time, the XML sitemap will become standard for all website development; therefore, additional pages, multimedia files or blogs will be easier to add to a site. Better still, any updates to a site will be seen by all three search engines, therefore eliminating the need to update this information three times to cater to three different search engines.
Ask.com has also begun to support XML sitemaps. A recent update to the XML Sitemap Protocol has made it possible for Ask.com to “tell” search engines the location of a site’s sitemap via placement of an entry into the robots.txt file.
When Should You Use Sitemaps?
Usually, if web pages are properly linked, search engine crawlers are able to discover most parts of the site. However, if you have a sitemap, it makes crawling easier and more efficient. Let’s look at some instances where you need sitemaps:
- When your site is quite large - there is a possibility that search engine spiders might overlook elements of your site, especially recently updated pages or new pages.
- When your site is new, with few external links to it - search engine bots crawl the web by following links from one page to another. When your site is new and has no links, search engines might not crawl your pages at all.
- When there is a large archive of content pages which are isolated or not properly linked - you need to list pages that do not naturally link to each other on a sitemap.
- You are using sitemaps-compatible annotations (rich media content or content that is shown in Google News) - Google is able to take the additional information from the sitemaps into account and display it in search results.
It is important to note that just because you have a sitemap does not mean that all the items listed will actually be crawled and indexed. However, there are still many benefits of having a sitemap, which we have already discussed.
Creating a Sitemap
Let’s look at an image of a site map before we begin:
It looks quite complicated, doesn’t it? The good news is that you do not have to know how to code to implement a sitemap. We will look at different options for creating sitemaps below:
Using a Content Management System
If you use a Content Management System like WordPress, you can generate sitemaps automatically via the help of plugins like Yoast SEO.
Adding a Sitemap to WordPress
We will focus on how to add a sitemap to WordPress via Yoast SEO. When you use this plugin, the sitemap index is updated automatically as you add or remove content. The post types that you want to be indexed are also included. Note that post types marked “noindex” will not appear in the sitemap.
To create your sitemap using Yoast:
- Log in to your WordPress account and access your dashboard.
- Click on “SEO” on the left-hand side menu.
The SEO settings expand to give you more options.
- Choose “general.”
- Click on features.
- Toggle the “XML sitemaps” to switch to “on.”
- Save changes.
This will activate the XML sitemap.
Viewing the Sitemap
- Log in to your WordPress account and access the dashboard.
- Click on “SEO” on the left-hand side menu.
The SEO settings expand to give you more options.
- Click “general” then select the features tab.
- To view your sitemap, click the question mark that is next to the XML sitemaps toggle
Using All in One SEO, you’ll need to go to All in One SEO > Feature Manager and Activate the XML Sitemaps:
The XML Sitemap will then appear in the All in One SEO settings:
Depending on your site needs, you can click on the help ? icon to learn more on what you can do for each settings. We recommend leaving the check on for:
- Create compressed sitemap
- Link from Virtual Robots.txt
- Dynamically generate sitemap
Where is Sitemap for Wix?
Wix sites have sitemaps that are dynamically generated on their server, and they automatically update when pages or content is added or removed from your site. These sitemaps are not editable.
Do I Need to Generate a Sitemap for Shopify?
All Shopify stores have an automatically generated sitemap.xml file. The file contains links to all products, pages, blog posts, product images, and collections. You’ll need to locate your sitemap file at the root directory of your store’s primary domain name (for example, suescollection.com/sitemap.xml) and then submit it to Google Search Console. We will look at this process a bit later in the article.
Using Other Tools to Generate a Sitemap
You can validate your current sitemap by using our sitemap validator tool. You can see if your sitemap has any errors to determine if you need to generate a new one:
To help you generate a sitemap, here are some tools that you can use, including SEOptimer Sitemap Generator and XML-sitemaps.com. Let’s take a look at how you can use both:
Using XML-sitemaps.com
- Go to https://www.xml-sitemaps.com/
- Copy your website URL
- Paste it on the website
- Click “start”
- Wait while your website is being crawled. Note: that if your website has more than 500 URLs, you will be required to create a paid account.
- Once the crawler is done, click “view sitemap details” then download the .XML file
At this point, it helps to review your sitemaps as any problem on your site is summarized. You may, for instance, find an issue like broken links. Let’s explore this a bit. Broken links can actually prevent your website from being totally indexed, as search engines see them as dead ends. Broken links suggest there is no more information, while in essence, the information is just inaccessible. Broken links can also be detrimental to a site’s reputation and increase your bounce rates.
- After reviewing your sitemaps for errors, you need to upload the sitemap to the domain root folder of your website.
- Next, open the Google Search Console and add your sitemap URL .
Using SEOptimer Sitemap Generator
- To generate a sitemap, you’ll need to enter your website:
- Whether you are modifying or creating a new sitemap,
- You may also need to add information, such as the last modified date, the frequency that your pages are likely to change, and the priority of your URLs.
- Select the number of pages your page has (up to 5,000)
- Hit the “Generate Sitemap” button.
- You can see that the crawler has finished crawling your site:
- You can either click Save XML file or copy to Clipboard and add to your current or upload your newly created sitemap XML file.
You'll need to validate your sitemap using SEOptimer Sitemap Validator, once you uploaded your sitemap to your website. You then need to submit your sitemap to Google Search Console. Submitting to Bing Webmaster Tools is also a good idea. Let's look at how to do both of these next:
How to Add a Sitemap to Google Search Console
You have just uploaded your sitemap to your root directory. The location of your sitemap should be https://mywebsite.com/sitemap.xml.
To upload it to Google Search Console, you need to log in to Search Console, go to “Index” then select “sitemaps.” Click “add/test Sitemap.” Enter the URL of the sitemap. Click “Submit”
How to Add a Sitemap to Bing Webmaster Tools
To register your sitemap with Bing Webmaster Tools, confirm that the XML sitemap that you uploaded to your site was indeed successful and that the URL loads.
- Log in to Bing Webmaster Tools
- Click “Configure My Site”
- Click “sitemaps”
- Enter your sitemap URL in the text box labeled “submit a sitemap”
- Click “submit”
Best Practices for Creating a Sitemap?
1. Always Test Your Sitemaps before Submission
Always test your sitemap before you submit it to ensure that any errors are resolved.
Here is how to test your sitemap:
- Click “add/test sitemap” on the reporting landing page on Google Search Console.
- Enter the URL of the sitemap in the dialog box and click Test.
- Click “Open Results” once the test is complete to check for any errors.
- You can then submit.
2. Prioritize High-Quality Pages in Your Sitemap
Your sitemap should not direct search engine bots to low-quality pages as it could be considered a sign of a site with minimal value to visitors. Prioritize highly optimized pages, images, videos, unique content, and other pages that prompt user engagement on your sitemap.
3. Only Change Modification Times when You have Made Some Substantial Changes
A black-hat SEO technique is to change modification times in order to trick search engine bots into thinking that you have updated your site significantly. Google may remove your date stamps if you do this.
4. Use Only Canonical Versions of URLs in Your Sitemap
If you have used the rel=canonical tag to differentiate pages that are very similar, then only include these on your sitemap. If you have not used the tag, you need to use it to distinguish pages that are similar (e.g., product pages) and only use the canonical version in your sitemap moving forward.
5. Keep Your Sitemap File Small
When your sitemap is small, it will not strain your server. The size of sitemaps was increased to 50 MB by Google and Bing.
6. Use Robots.txt for Pages that You do not Want Indexed
Using a sitemap does not mean that you need to index each and every page. Pages such as thank you pages should not be indexed even if they are listed on the sitemap. You should use the meta robots “noindex, follow” tag to preserve your link equity even if that particular page will not be indexed.
There are times when you can use robots.txt to block pages. One such instance is when your crawl budget is quickly used up. What is a crawl budget? It is the number of times a search engine bot crawls your site within a particular time frame. If your site gets crawled 32 times a day, for instance, there is a possibility that you have a monthly crawl budget of 960 times.
You can see your crawl budget under “crawl stats” in the search console.
7. If You have More than 50,000 URLs, Create Multiple Sitemaps
A sitemap has a limit of 50,000 URLs. This is adequate for most sites. If you have a large site, such as e-commerce sites, you will need to create more than one sitemap.
8. Avoid Including “noindex” URLs in Your Sitemap
If you do not want certain pages to be crawled, it makes more sense to avoid including them in the sitemap rather than adding a “noindex” tag. This tag sends out conflicting information to the search engine bot, suggesting that the page should both be and not be indexed. This inconsistency wastes your crawl budget.
9. If You have a Large Site, Create Dynamic Sitemaps
If your site is quite large, you need to set up some rules that will help determine when a page can be included in your sitemap, or changed from “noindex” to “indexfollow.” It may help to use a tool to generate a dynamic XML Sitemap.
10. Combine XML Sitemaps and RSS/Atom Feeds
RSS/Atom Feeds help with notifying search engines when content is updated on your site and make it easier for both search engines and users to access fresh content.
11. Other tips
- Ensure that your site structure flows from left to right, as people intuitively click from left to right.
- Use simple names for your pages, such as “products” and “services.”
- Your menu needs to appear on every page.
- Your menu needs to be at the top of the page and list the main navigation page.
- Use subcategories so that your menu stays simple.
In Conclusion
Sitemaps make it easier for search engines to index and crawl your site. They also make it easier for users to navigate your site. This way, when you update content, both search engines and users will easily find the new material. What does this mean? That you are more likely to have a good ranking and high sight reputation, resulting in more and new visitors as well as conversions.
