llms.txt and AI crawlers
Generative AI tools read the web through their own crawlers. RankAIZI helps you welcome them and see them.
What is llms.txt?
It is a plain file at /llms.txt that describes your site for AI tools. In Settings, turn on llms.txt and RankAIZI serves it. It is off by default.
How do I welcome AI crawlers?
Turn on the setting that allows AI crawlers and RankAIZI adds rules for GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot and Applebot-Extended to robots.txt. It is off by default. You can also insert the block from the Files screen.
Can I see which AI crawlers visit my site?
Yes. The AI crawler log counts visits per day by crawler name, with the last time and last address seen. It stores no IP address. It is on by default and can be turned off in Settings.
Related guides
Still stuck? Ask the assistant in the chat bubble, or write to info@ainextstudio.it.