How to create a sitemap or urllist and robots files ?
The sitemap or urllist is a txt file which helps web search tools to index your website more easily and by this way, to let know to the whole world that you exist !!! The robots is a txt file which allows to block or to unblock web search tools to search within all your website or only inside some parts. I will say that the sitemap/urllist is mandatory not only to let know your existence of the world but also to attract visitors in order to increase your traffic. The robots will be useful only if you have something to hide and that you don’t want google, yahoo... to search in some sections for example if you have a download folder. |
|
Robots.
In theory and in practice, if you have nothing to hide or to protect, you don’t need to create a robots file to tell web search tools to index your websi<te because they will do it automatically if not read the following instruction.
To create a robots file, open notepad and you have to call it robots.txt – don’t save it like robotsmysite.txt or other names because web search tools will not recognize it.
An unblocked file will contain this information:
User-Agent: *
Allow: /
And a blocked one will contain this:
User-Agent: *
Disallow: /
User-agent means that you allow all kind of web search tool agents to search in your website. Allow will tell those agents that they can look inside the whole domain and Disallow will tell the contrary.
Let’s say that you have a download folder that you want to forbid yahoo and google so by this way, people who tape “download” like search word, both will search everywhere except inside this folder.
User-Agent: *
Disallow: /download/
Now let’s say that you want to block all your website but you will allow only some sections and some files.
User-Agent: *
Allow: /sample1.html
Allow: /sample2.html
Allow: /sample1/
Allow: /sample2/
Disallow: /
If you want more information, you can check the Robots website.
Sitemap or urllist.
There are no difference for both but if you want yahoo to index your website, you have to name your file urllist.txt and if you want google to index your website, you have to name it sitemap.txt. The contents for both are the same and don’t save them with another name.
Your sitemap/urllist should contain every files in your website, well at least all the files that you want web search tools to index. More files, more indexed pages and more possibility for people to discover your website and by this way to increase your traffic.
To create them, you have to open notepad and put the information with the full url like following:
https://www.world-rt.com/
https://www.world-rt.com/blog/silent-hill-homecoming.php
If you have too much files, you can use like me the free service of xml-sitemaps.com. If you have created a robots.txt file to block your website, don’t forget to disable it just for the time to create your sitemap.txt / urllist.txt files if not you will get a blank file !!!
After upload the 3 files in the root of your domain and don't forget to tell to the web search tools that you have those files.