feat(seo): site config, sitemap, robots.txt, llms.txt and a real 404 page
- astro.config: site, trailingSlash "never", build.format "file". - /sitemap-index.xml + /sitemap-0.xml as prerendered endpoints (no new dependency); pages are derived from src/pages/*.astro, 404 excluded. - robots.txt: all bots allowed, AI crawlers/training bots explicitly allowed (Linus's decision), /download/ disallowed, sitemap line. - /llms.txt generated from business.ts, incl. usage-with-attribution note. - 404.astro in Swedish with links back, noindex. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WvE3C43Uwtv2vtnoFBLDw7
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
30e385ec7f
commit
f452d6d237
@@ -0,0 +1,41 @@
|
||||
# robots.txt – https://imagesbyolofsson.se
|
||||
#
|
||||
# Policy (beslut av Linus 2026-10-27): ALLA botar är välkomna, även AI-crawlers
|
||||
# och träningsbotar. Innehållet får användas av AI-assistenter med hänvisning
|
||||
# till Images by Olofsson (se /llms.txt). Cloudflare-sidan (AI Crawl Control /
|
||||
# "Block AI bots" / hanterad robots.txt) ställs in separat så att den inte
|
||||
# blockerar eller skriver över detta.
|
||||
|
||||
User-agent: *
|
||||
Allow: /
|
||||
# Lösenordsskyddad nedladdning av originalfiler – inget att indexera.
|
||||
Disallow: /download/
|
||||
|
||||
# AI-crawlers och träningsbotar: uttryckligen tillåtna.
|
||||
User-agent: GPTBot
|
||||
User-agent: OAI-SearchBot
|
||||
User-agent: ChatGPT-User
|
||||
User-agent: ClaudeBot
|
||||
User-agent: Claude-User
|
||||
User-agent: Claude-SearchBot
|
||||
User-agent: anthropic-ai
|
||||
User-agent: CCBot
|
||||
User-agent: Google-Extended
|
||||
User-agent: Applebot
|
||||
User-agent: Applebot-Extended
|
||||
User-agent: Bytespider
|
||||
User-agent: Amazonbot
|
||||
User-agent: meta-externalagent
|
||||
User-agent: meta-externalfetcher
|
||||
User-agent: FacebookBot
|
||||
User-agent: PerplexityBot
|
||||
User-agent: Perplexity-User
|
||||
User-agent: cohere-ai
|
||||
User-agent: Diffbot
|
||||
User-agent: YouBot
|
||||
User-agent: DuckAssistBot
|
||||
User-agent: MistralAI-User
|
||||
Allow: /
|
||||
Disallow: /download/
|
||||
|
||||
Sitemap: https://imagesbyolofsson.se/sitemap-index.xml
|
||||
Reference in New Issue
Block a user