feat(seo): site config, sitemap, robots.txt, llms.txt and a real 404 page

- astro.config: site, trailingSlash "never", build.format "file".
- /sitemap-index.xml + /sitemap-0.xml as prerendered endpoints (no new
  dependency); pages are derived from src/pages/*.astro, 404 excluded.
- robots.txt: all bots allowed, AI crawlers/training bots explicitly
  allowed (Linus's decision), /download/ disallowed, sitemap line.
- /llms.txt generated from business.ts, incl. usage-with-attribution note.
- 404.astro in Swedish with links back, noindex.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WvE3C43Uwtv2vtnoFBLDw7
This commit is contained in:
Linus Olofsson
2026-10-27 19:13:07 +01:00
co-authored by Claude Opus 5.5
parent 30e385ec7f
commit f452d6d237
7 changed files with 198 additions and 0 deletions
+41
View File
@@ -0,0 +1,41 @@
# robots.txt – https://imagesbyolofsson.se
#
# Policy (beslut av Linus 2026-10-27): ALLA botar är välkomna, även AI-crawlers
# och träningsbotar. Innehållet får användas av AI-assistenter med hänvisning
# till Images by Olofsson (se /llms.txt). Cloudflare-sidan (AI Crawl Control /
# "Block AI bots" / hanterad robots.txt) ställs in separat så att den inte
# blockerar eller skriver över detta.
User-agent: *
Allow: /
# Lösenordsskyddad nedladdning av originalfiler – inget att indexera.
Disallow: /download/
# AI-crawlers och träningsbotar: uttryckligen tillåtna.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: FacebookBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: cohere-ai
User-agent: Diffbot
User-agent: YouBot
User-agent: DuckAssistBot
User-agent: MistralAI-User
Allow: /
Disallow: /download/
Sitemap: https://imagesbyolofsson.se/sitemap-index.xml