In other words, they’re assholes.
The only surprising thing to me from this article is that OpenAI actually follows the rules for bot crawlers.
Or they haven’t been caught yet.
The article explains PerplexityBot respects robots.txt, but then sends a different request with a different IP and different user-agent. They could very well be using a different method to walk around it.
The article explains how they tested for that, and as far as they could tell OpenAI is respecting the rules.
A sure sign that they are a nefarious company.
Perplexity fired back in their blog.
Pretty tasty.
Perplexity’s firing back assumes website owners distinguish between automated scraping and on-demand scraping.
I don’t think most people make that distinction.
And that falls in line perfectly with the typical “assumption of access” all of these “AI” companies make.
