Pro@mander.xyz to

Technology@programming.devEnglish · 1 month ago

Cloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocks

blog.cloudflare.com

7

cross-posted to:
technology@lemmy.zip

55

Cloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocks

blog.cloudflare.com

Pro@mander.xyz to

Technology@programming.devEnglish · 1 month ago

7

cross-posted to:
technology@lemmy.zip

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

blog.cloudflare.com

Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.

Perplexity blog.

Chat

beeng@discuss.tchncs.de
link
fedilink
English
arrow-up
3·
1 month ago
Perplexity fired back in their blog.

Pretty tasty.

Technology@programming.dev

Technology@programming.dev

You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !Technology@programming.dev

Share interesting Technology news and links.

Rules:

No paywalled sites at all.
News articles has to be recent, not older than 2 weeks (14 days).
No external video links, only native(.mp4,…etc) links under 5 mins.
Post only direct links.

To encourage more original sources and keep this space commercial free as much as I could, the following websites are Blacklisted:

Al Jazeera;
NBC;
CNBC;
Substack;
Tom’s Hardware;
ZDNet;
TechSpot;
Ars Technica;
Vox Media outlets, with exception for Axios;
Engadget;
TechCrunch;
Gizmodo;
Futurism;
PCWorld;
ComputerWorld;
Mashable;
Hackaday;
WCCFTECH;
Neowin.

More sites will be added to the blacklist as needed.

Encouraged:

Archive links in the body of the post.
Linking to the direct source, instead of linking to an article talking about the source.

Misc:

Relevant Communities:

Beehaw Technology- Technology Related Discussions.
lemmy.zip Technology- Hard Tech news.

Visibility: Public

This community can be federated to other instances and be posted/commented in by their users.

378 users / day
2.13K users / week
4.92K users / month
7.14K users / 6 months
1 local subscriber
552 subscribers
1.6K Posts
3.53K Comments
Modlog

mods:
Pro@programming.dev