zydronium/neocities
1
0
Fork 0
mirror of https://github.com/neocities/neocities.git synced 2025-07-03 01:23:22 +02:00
Code Issues Projects Releases Packages Wiki Activity

remove Applebot from list (Applebot-Extended determins LLM usage)

Browse source
This commit is contained in:
Kyle Drake 2024-12-30 17:40:19 -06:00
parent 2cfc848ab2
commit bb30252000
1 changed files with 1 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

3
views/templates/robots.txt
View file

@ -9,7 +9,6 @@ Allow: /
#User-agent: Ai2Bot-Dolma
#User-agent: Amazonbot
#User-agent: anthropic-ai
#User-agent: Applebot
#User-agent: Applebot-Extended
#User-agent: Bytespider
#User-agent: CCBot
@ -46,4 +45,4 @@ Allow: /
#User-agent: VelenPublicWebCrawler
#User-agent: Webzio-Extended
#User-agent: YouBot
#Disallow: /
#Disallow: /