Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 8 个月前How to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square62fedilinkarrow-up1111arrow-down132
arrow-up179arrow-down1external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 8 个月前message-square62fedilink
minus-squaremox@lemmy.sdf.orglinkfedilinkarrow-up26·8 个月前This article lies to the reader, so it earns a -1 from me.
minus-squareCynicus Rex@lemmy.mlOPlinkfedilinkarrow-up6arrow-down8·edit-28 个月前Lies, as in that it’s not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out?
minus-squareDa Bald Eagul@feddit.nllinkfedilinkarrow-up37·8 个月前That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.
minus-squareJackbyDev@programming.devlinkfedilinkEnglisharrow-up4arrow-down4·8 个月前Is it a lie or a simplification for beginners?
minus-squarethanks_shakey_snake@lemmy.calinkfedilinkarrow-up13·8 个月前Lie. Or at best, dangerously wrong. Like saying “Crosswalks make cars incapable of harming pedestrians who stay within them.”
minus-squareJackbyDev@programming.devlinkfedilinkEnglisharrow-up1arrow-down5·8 个月前It’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
minus-squarethanks_shakey_snake@lemmy.calinkfedilinkarrow-up3·8 个月前Maybe? But it’s not like that’s the only alternative thing to say, lol
minus-squareReversalHatchery@beehaw.orglinkfedilinkEnglisharrow-up2arrow-down1·edit-28 个月前Is it, though? I mean, robots.txt is the Do Not Track of the opposite side of the connection.
minus-squaremox@lemmy.sdf.orglinkfedilinkarrow-up4·8 个月前Assuring someone that they have control of something and the safety that comes with it, when in fact they do not, is well outside the realm of a simplification. It’s just plain false. It can even be dangerous.
minus-squareEager Eagle@lemmy.worldlinkfedilinkEnglisharrow-up1·8 个月前the word disallow is right there
This article lies to the reader, so it earns a -1 from me.
Lies, as in that it’s not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out?
That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.
Is it a lie or a simplification for beginners?
Lie. Or at best, dangerously wrong. Like saying “Crosswalks make cars incapable of harming pedestrians who stay within them.”
It’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
Maybe? But it’s not like that’s the only alternative thing to say, lol
Is it, though?
I mean, robots.txt is the Do Not Track of the opposite side of the connection.
Assuring someone that they have control of something and the safety that comes with it, when in fact they do not, is well outside the realm of a simplification. It’s just plain false. It can even be dangerous.
the word disallow is right there