{"id":1840,"date":"2021-05-13T06:35:08","date_gmt":"2021-05-13T13:35:08","guid":{"rendered":"https:\/\/www.ssls.com\/blog\/?p=1840"},"modified":"2023-12-06T07:49:47","modified_gmt":"2023-12-06T15:49:47","slug":"why-your-site-should-have-robots-txt-sitemap-xml","status":"publish","type":"post","link":"https:\/\/www.ssls.com\/blog\/why-your-site-should-have-robots-txt-sitemap-xml\/","title":{"rendered":"Why your site should have robots.txt & sitemap.xml"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Have a website? Want it to get found in search engines? If the answer to these questions is \u201cYes\u201d, then it\u2019s likely you\u2019ve given search engine optimization (SEO) a go at some point, or at least more than a passing thought. While the impulse when getting started with SEO is often diving headfirst into keyword research and link building (where spelling is super important; you should know the difference between affect and effect<\/a>, for example), there are some lesser talked about aspects of SEO that are just as important. Namely, two crucial files you should have on your site: robots.txt and sitemap.xml.
<\/p>\n\n\n\n\n\n\n\n

But what do these files do, and why are they so important? Read on to find out what they are, what they do, and how to add both to your site.
<\/p>\n\n\n\n

Before discussing robots.txt and sitemap.xml, you\u2019ll first need to understand two relevant terms: website indexing and web crawling. 
<\/p>\n\n\n\n

Web indexing<\/strong> is how search engines store and organize information about web pages across the World Wide Web. Essentially, indexing is the entire point of search engines! Where a website ranks in that index is dependent on a whole slew of SEO factors, from the aforementioned keywords to relevance and content quality.
<\/p>\n\n\n\n

Crawling <\/strong>is how search engines find pages across the web to index. Basically, each search engine has bots<\/a> known as crawlers that \u201ccrawl\u201d the web, seeking out new content or web pages to save to its index by following the links featured on each page they find. 
<\/p>\n\n\n\n

Now that you\u2019re an expert on all things indexing and crawling, let\u2019s move on to what you\u2019re here for.<\/p>\n\n\n\n

What are robots.txt & sitemap.xml?<\/h2>\n\n\n\n

Robots.txt and sitemap.xml are essential files that can help search engines better understand your particular website and index it correctly. For this reason, robots.txt and XML sitemaps go hand in hand. <\/p>\n\n\n\n

The importance of XML sitemaps<\/h3>\n\n\n\n

An XML sitemap is a blueprint of what you consider the most important parts of your website. While the name \u201csitemap\u201d might suggest an illustrated layout of your site, it\u2019s actually just a list of page links. Although web crawlers should be able to find the pages on your site well enough if they are properly linked (both internally and externally), an XML sitemap ensures that they will crawl and index the content you consider most pertinent and not, say, tag pages or a now irrelevant blog post from five years ago. 
<\/p>\n\n\n\n

XML sitemaps aren\u2019t mandatory, but they are valuable tools, particularly if you have a large website with many pages or \u2014 on the other end of the spectrum \u2014 a relatively new site that doesn\u2019t have many external links yet. 
<\/p>\n\n\n\n

You have the option of submitting your sitemap directly to search engines, but crawlers will be able to find it when they visit your site if you have a robots.txt file directing them to it.
<\/p>\n\n\n\n

What robots.txt does<\/h3>\n\n\n\n

A robots.txt file is a file that you can place in your website\u2019s root directory to instruct crawlers how you want your site to be crawled. These instructions can include which pages you want them to crawl, which ones they should avoid, or instructions to block specific bots from crawling the site entirely. When crawlers visit a site, it\u2019s the robots.txt file they usually visit first. It\u2019s also where you should place your XML sitemap location so the crawlers can easily find it.<\/p>\n\n\n\n

How to create an XML sitemap<\/h2>\n\n\n\n

So, now that you\u2019re up to speed on the necessity of XML sitemaps, how do you go about creating one? For the more technically inclined who want to make one manually, Google has instructions<\/a> on doing just that. There are also several free generators online, such as this one<\/a>.
<\/p>\n\n\n\n

For WordPress users, numerous plugins will create sitemaps for you, such as Yoast<\/a> and All in One SEO<\/a> (both of which are also pretty nifty general SEO tools).<\/p>\n\n\n\n

How to create and add a robots.txt file to your site<\/h2>\n\n\n\n

This can be a little tricky if you don\u2019t have access to your website\u2019s server. For WordPress sites, many sitemap plugins (such as the ones mentioned above) will do it for you. If you want to do it yourself, learn out how to create and upload a robots.txt file to your server with this handy how-to<\/a>. When you\u2019re done with that, check out this guide from Google<\/a> explaining how to add your XML sitemap link to the file.<\/p>\n\n\n\n

Conclusion<\/h2>\n\n\n\n

Robots.txt and XML sitemaps may not be at the top of your SEO considerations,\u00a0 but they shouldn\u2019t be overlooked. By taking the time to create a sitemap and adding a robots.txt file to your site, you\u2019ll have more of a say in how your website is crawled and ultimately indexed, which should have a positive impact on your overall SEO.<\/p>\n\n\n\n

While we have you, another great way of improving your site\u2019s SEO is encryption, so don\u2019t forget to check out our range of SSL certificates<\/a> for securing your site.
<\/p>\n","protected":false},"excerpt":{"rendered":"

Have a website? Want it to get found in search engines? If the answer to these questions is \u201cYes\u201d, then it\u2019s likely you\u2019ve given search engine optimization (SEO) a go at some point, or at least more than a passing thought. While the impulse when getting started with SEO is often diving headfirst into keyword […]<\/p>\n","protected":false},"author":9,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1840","post","type-post","status-publish","format-standard","hentry","category-news"],"_links":{"self":[{"href":"https:\/\/www.ssls.com\/blog\/wp-json\/wp\/v2\/posts\/1840","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ssls.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ssls.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ssls.com\/blog\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ssls.com\/blog\/wp-json\/wp\/v2\/comments?post=1840"}],"version-history":[{"count":4,"href":"https:\/\/www.ssls.com\/blog\/wp-json\/wp\/v2\/posts\/1840\/revisions"}],"predecessor-version":[{"id":2840,"href":"https:\/\/www.ssls.com\/blog\/wp-json\/wp\/v2\/posts\/1840\/revisions\/2840"}],"wp:attachment":[{"href":"https:\/\/www.ssls.com\/blog\/wp-json\/wp\/v2\/media?parent=1840"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ssls.com\/blog\/wp-json\/wp\/v2\/categories?post=1840"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ssls.com\/blog\/wp-json\/wp\/v2\/tags?post=1840"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}