How to block AI Crawler Bots using robots.txt file
IphtashuFitz @ IphtashuFitz @lemmy.world Posts 3Comments 885Joined 2 yr. ago
IphtashuFitz @ IphtashuFitz @lemmy.world
Posts
3
Comments
885
Joined
2 yr. ago
Deleted
Thanks, Grok.
Exactly. The only truly effectively way I’ve ever found to block bots is to use a service like Akamai. They have an add-on called Bot Manager that identifies requests as bots in real time. They have a library of over 1000 known bots and can also identify unknown bots built on different frameworks, bots that impersonate well known bots like Googlebot, etc. This service is expensive, but effective…