how to customize robots.txt file


What Is Robots.txt SEO and How to use it In blogger - Detailed Explanation of robots.txt File


Robots.Txt



Guys, Do you Love, "."

Sure, you Do That`s Why you are here,

So, Today I am gonna talk about legitimate SEO Tip that can help you increase your traffic right away; just kidding, there are no tip that can increase your traffic right away,

But this tip can help you a lot, and there is no rocket-science behind implementing it.


It’s designed to work with search engines, but surprisingly, it’s a source of SEO juice waiting to be unlocked.


There are too many methods for improving SEO without requiring any technical experience; neither are they difficult to implement nor time-consuming.

So, when you are ready, get along with me; I'll show you how to change your robots.txt file perfectly so that your website get loved by google.


What Is Robots.txt File?


A robot.txt file (also known as the robots exclusion protocol or standard
) is a tiny file that contains a set of texts that tell search engine crawlers what page/post they want to index what page to index(crawl).  and which pages not to  indexed(Crawl)

Blogger, if you do not add a custom robot.txt file, Blogger uses the default robot.txt file that can be modified.

The robots.txt file can be accessed for any site by adding another robots.txt site domain.


the file we use in this blog can be accessed from 

https://www.stealprice.online/robots.txt


Let`s Say when a search engine about to visit your page, it will check the robots.txt file instruction before crawling.


let`s say a Search engine find this type of robots.txt on your website

the basic skeleton of a robots.txt file.
the basic skeleton of a robots.txt file.

This Is a very Basic Types of robots.txt

Let`s Understand what is says


The asterisk(*) after “user-agent” means that the robots.txt file applies to all web robots that visit the site.


The slash after “Disallow” means that the robot will not visit any pages (Not Posts) on the site.

If you want search engine bots to crawl specific posts or content, the bots will crawl and index your site based on that content alone.


As Google Says:


“You don’t want your server to be overwhelmed by Google’s crawler or to waste crawl budget crawling unimportant or similar pages on your site.”


Components Of Robots.txt file 


User-agent

It determines the type of crawler to which the commands will be given. To select the Facebook crawler facebookexternalhit / 1.1, 

for example, we will use these lines.


User-agent: facebookexternalhit/1.1

Disallow: /p

Disallow: /search/label

Allow: /


User-agent: *

Allow: /


How to Customize the robots.txt file


To customize your robots.txt file for your blog, log in to Blogger Settings Search preferences ⇽ Indexing and crawlers Custom robots.txt file and click Edit then select Yes and finally add the lines you want to use

how to customize robots.txt file



Complete the explanation for more information on customizing the file


The default robots.txt file


If you don't customize the robots.txt file, then Blogger uses this:



User-agent: Mediapartners-Google


Disallow:




User-agent: *


Disallow: /search


Allow: /




Sitemap: https://www.stealprice.online/sitemap.xml



This file contains important information that directs the reptiles; there are different types of reptiles, including search spiders(crawler) that specialize in searching for topics to archive, and there are crawlers specialized in fetching information such as facebookexternalhit / 1.1 of Facebook and its function is to crawl the links that have been shared on Facebook to bring pictures and information Stored in it or the Alexa crawler ia_archiver whose function is to browse sites for content analysis, related sites, and other analytical functions ...


There are many types of crawlers, each crawler and the function developed to perform it.


Robots.txt file components


User-agent


It determines the type of crawler to which the commands will be given. To select the Facebook crawler facebookexternalhit / 1.1, for example, we will use these lines.


User-agent: facebookexternalhit/1.1


Disallow: /p


Disallow: /search/label


Allow: /


User-agent: *


Allow: /


Sitemap: https://www.stealprice.online/sitemap.xml



The commands specified in red will be applied to the Facebook crawler only - the commands are explained below - but if we use the * sign in the user agent then specify all other crawlers without the specified in the code, the command specified in green means allow: / found in the user-agent * will not work on the Facebook crawler Because our Facebook crawler has assigned its own commands alone.


User-agent commands


Disallow -


You can add a Disallow line to prevent a crawler from accessing all or some of your blog pages.


Example:


User-agent: *


Disallow: /


This line will block all kinds of crawlers from accessing your site.


* His job is to identify all kinds of reptiles - Note


And Disallow prevents access to any page on the site that starts with / means all pages; if you want to prevent access to some pages, you can specify it like this example.


User-agent: *


Disallow: /p


Disallow: /2018/03/something.html


Disallow: /search


Thus, crawlers will not be able to access the mentioned pages, and these are examples of some links that cannot be reached when using these lines:


https://www.stealprice.online/p/contact.html


https://www.stealprice.online/2018/03/something.html


https://www.stealprice.online/search/label/something


https://www.stealprice.online/search?q=something


Either if Disallow is empty like:


User-agent: *


Disallow: 


Crawlers will access all of your pages.


Allow -


You can add an Allow line to allow the crawler to access specific pages; for example, use.



User-agent: *


Disallow: /p


Allow: /p/important.html


Disallow prevents access to any page in the blog because the Blogger hosts all the pages on links beginning with / p,


except that Allow allowed one page, which is important.  


Sitemap


It tells the search spiders where to store the map of your blog. To facilitate access to blogging links in Blogger, the blog map can be accessed in different ways.


https://www.stealprice.online/feeds/posts/default?orderby=UPDATED


https://www.stealprice.online/feeds/posts/default?alt=atom


https://www.stealprice.online/sitemap.xml


https://www.stealprice.online/atom.xml


From my perspective, the best is the first because it brings topics in the order of the last update, meaning any topic you updated becomes at the top of the map.


You add it at the bottom of the robots.txt file like this.



User-agent: *


Allow: /



Sitemap: https://www.stealprice.online/feeds/posts/default?orderby=UPDATED


Comments #


It is also possible in robots.txt files to add comments as in various programming languages, you may want to add comments to remember something or create an alert about why you added a command or just as a note... To


write a comment, start with # at the beginning of the speech as in the example


User-agent: *


Allow: /


# The page below has been denied access because it contains important information that we do not want to archive


# this page contains sensitive info


Also, Read,


How to get approved for AdSense in 2020| get approved 100%

What Is Blogger Cleanup Code: How To Install New Template in Blogger Easily in 2020



Disallow: /p/secret.html


Sitemap: https://www.stealprice.online/ feeds / posts / default? orderby = UPDATED



How to use the wildcard in links


The wildcard * helps you to define a set of infinite characters.


The wildcard can mean "everything," for example:


User-agent: *


# In the code below, we have blocked access to the entire blog


Disallow: /


#, but we have enabled access to a group of other pages.


Allow: / p / * .html



We made all pages accessible because * can mean everything, a line can define an unlimited number of pages, and these are some examples.


https://www.stealprice.online/p/contact.html


https://www.stealprice.online/p/usage-agreement.html


https://www.stealprice.online/p/privacy-policy.html



We come to the end of the explanation. If there are questions about the robots.txt file, you can ask it in a comment.


I hope the topic is as understandable as possible.