Microsoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Reddit
DaGeek247 @ DaGeek247 @fedia.io Posts 5Comments 264Joined 1 yr. ago

DaGeek247 @ DaGeek247 @fedia.io
Posts
5
Comments
264
Joined
1 yr. ago
My robots.txt has been respected by every bot that visited it in the past three months. I know this because i wrote a page that IP bans anything that visits it, and l also put it as a not allowed spot in the robots.txt file.
I've only gotten like, 20 visits in the past three months though, so, very small sample size.