Pro@programming.dev to Technology@programming.devEnglish · edit-26 months agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comimagemessage-square116linkfedilinkarrow-up1708arrow-down12file-text
arrow-up1706arrow-down1imageCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.dev to Technology@programming.devEnglish · edit-26 months agomessage-square116linkfedilinkfile-text
minus-squarewho@feddit.orglinkfedilinkEnglisharrow-up2·7 months ago Crawl-delay It’s a nonstandard extension without consistent semantics or wide support, but I suppose it’s good to know about anyway. Thanks for mentioning it.
It’s a nonstandard extension without consistent semantics or wide support, but I suppose it’s good to know about anyway. Thanks for mentioning it.