mikaela.github.io/robots.txt

User-agent: *
Allow: /$
Allow: /*.html$
Allow: /blog/$
Allow: /blog/feed.xml
Allow: /blog/feed.en.xml
Allow: /blog/feed.fi.xml
Allow: /irc/$
Allow: /matrix/$
Allow: /n/$
Allow: /robots.txt
Allow: /sitemap.xml
Allow: /sitemaps.xml
Disallow: /
Disallow: /*.asc$
Disallow: /*.json$
Disallow: /*.sig$
Disallow: /*.txt$
Disallow: /*.toml$
Disallow: /*.xml$
Disallow: /txt/
Sitemap: https://aminda.eu/sitemap.xml
Sitemap: https://aminda.eu/sitemaps.xml
add robots.txt 2015-02-06 07:39:24 +01:00			`User-agent: *`
robots.txt: add more explicit allow rules while disallowing xml 2024-06-02 11:35:16 +02:00			`Allow: /$`
robots.txt: even more specific rules 2024-06-02 11:49:59 +02:00			`Allow: /*.html$`
robots.txt: add more explicit allow rules while disallowing xml 2024-06-02 11:35:16 +02:00			`Allow: /blog/$`
			`Allow: /blog/feed.xml`
			`Allow: /blog/feed.en.xml`
			`Allow: /blog/feed.fi.xml`
			`Allow: /irc/$`
			`Allow: /matrix/$`
			`Allow: /n/$`
robots.txt: even more specific rules 2024-06-02 11:49:59 +02:00			`Allow: /robots.txt`
robots.txt: add more explicit allow rules while disallowing xml 2024-06-02 11:35:16 +02:00			`Allow: /sitemap.xml`
			`Allow: /sitemaps.xml`
robots.txt: even more specific rules 2024-06-02 11:49:59 +02:00			`Disallow: /`
robots.txt: disallow *.{asc,json,sig,txt,toml} 2024-06-02 11:17:35 +02:00			`Disallow: /*.asc$`
			`Disallow: /*.json$`
			`Disallow: /*.sig$`
			`Disallow: /*.txt$`
			`Disallow: /*.toml$`
robots.txt: add more explicit allow rules while disallowing xml 2024-06-02 11:35:16 +02:00			`Disallow: /*.xml$`
robots.txt: Disallow /txt/ and /php/ as they don't contain HTML noindex 2020-11-23 12:39:27 +01:00			`Disallow: /txt/`
more apex domain changes 2023-03-05 11:25:56 +01:00			`Sitemap: https://aminda.eu/sitemap.xml`
			`Sitemap: https://aminda.eu/sitemaps.xml`