Does anybody have recommendations for a proper robots.txt file? Any websites or tools that can help me write it? (not AI obviously)
Thanks for the follow, Caesár! I enjoyed your ballad, “I Wake Up, Take a Shower,” which I feel encapsulates the monotony of the daily grind. 😊
1 like
love the dutch NS beat, thanks for the follow!
1 like
1 like
Thanks for the follow! I love your site!
1 like
2 likes
I didn't realise newer Neocities sites automatic have a robots.txt file to prevent them from getting snooped by ChatGPT and stuff, are you supposed to leave the # on the last line or not?
1 like
win1999
11 months ago
This configuration is very effective at blocking AI bots, but it has a major drawback: it also blocks several services that are not just AI-related. Consider exemptions.
1 like
1 like
![xX [ TR/BZ ] Xx avatar](/site_screenshots/41/77/toribytez/index.html.50x50.webp)
![tekerare [dot] neocities [dot] org avatar](/site_screenshots/26/24/tekerare/index.html.50x50.webp)
![auberylis [dot] moe avatar](/site_screenshots/34/10/astrossoundhell/index.html.50x50.webp)
![[tfpXE] avatar](/site_screenshots/15/25/tfpxe/index.html.50x50.webp)
isn't there a basic one that everyone uses? unless that's not 'proper'
I'm not tech savvy on that level, mine wasn't correct but I don't know which crawls to block or allow
I'd actually reccomend checking out this page in specific: https://baccyflap.com/res/robots/. They have a great list for a robots.txt file, along with a few other measures to assist in bot proofing a site!