What it is
A small file at the site root that tells robots what they may and may not read. One wrong line in it can accidentally hide the whole site from search.
How we use it
Our robots.txt opens the site to all search engines and AI bots (GPTBot, ClaudeBot, Bingbot and others), closing only service addresses. In staging mode one variable hides the site from indexing.
Where it matters to a business
- Search engines and AI assistants should read the site, but not the admin panel and service URLs.
- A staging version must not end up in search instead of the real site.
How we use it
- Open to search and AI. Our robots.txt opens the site to all search engines and AI bots (GPTBot, ClaudeBot and others), closing only service URLs. On the studio website the file is generated by code, and one environment variable switches the site to staging mode with indexing disallowed.
- The admin panel is closed. In the online store robots.txt closes the admin panel.
- A demo closed entirely. For the museum site demo stand robots.txt is only one of three levels, alongside a meta tag and an X-Robots-Tag header that also covers images, video and fonts.
Common problems
- One line closes everything. A block left over from staging is a common reason a site is missing from search.
- robots.txt is a request, not a lock. Honest robots obey it, others do not. That is why the demo stand is also protected by a gate and refusals for known site downloaders.
When you do not need it
Every site needs robots.txt; the only question is what it closes. For real privacy you need a password, not robots.txt.