The internet is full of robots, and their ranks are growing exponentially. In 2015 roughly half of all internet traffic was bots, so it’s important to pay attention to how robots interact with your site. These robots are software that automatically browse the internet. The most common kind of robot crawls the internet and indexes content from search engines, but there are also bad bots, which harvest emails or search for sites that are weak against mass hacking attempts. Luckily, you can use a robots.txt file to communicate with all of these non-human netizens, and in some cases even give them instructions on how to use your site. Read more
Articles about robots.txt