LLMs.txt Generator
SYSTEM_ACCESS_CONTROL
OpenAI (ChatGPT)
// Primary web crawler for OpenAI models.
Google AI
// Opt-out crawler for Bard and Vertex AI models.
Anthropic (Claude)
// Crawler for the Claude family of models.
Cohere
// Crawler for Cohere language models.
OpenAI (Legacy)
// Another user-agent used by OpenAI.
Common Crawl
// Scrapes the web for a publicly available dataset.
ByteDance
// Web crawler from the owners of TikTok.
Meta AI
// Crawler for Facebook/Instagram AI features.
Apple
// Used for Siri and other Apple intelligence products.
Ahrefs
// SEO tool for backlink and site analysis.