AI companies are violating a basic social contract of the web and and ignoring robots.txt
ShitpostCentral @ ShitpostCentral @lemmy.world Posts 2Comments 28Joined 2 yr. ago
ShitpostCentral @ ShitpostCentral @lemmy.world
Posts
2
Comments
28
Joined
2 yr. ago
You're second point is a good one, but you absolutely can log the IP which requested robots.txt. That's just a standard part of any http server ever, no JavaScript needed.