Mikaela/mikaela.github.io
1
0
Fork 0
mirror of https://github.com/mikaela/mikaela.github.io/ synced 2024-11-22 12:09:28 +01:00
Code Issues Projects Releases Wiki Activity

robots.txt: add more explicit allow rules while disallowing xml

Browse Source
This commit is contained in:
Aminda Suomalainen 2024-06-02 12:35:16 +03:00
parent b87a66d59a
commit b23e622fae
Signed by: Mikaela
SSH Key Fingerprint: SHA256:CXLULpqNBdUKB6E6fLA1b/4SzG0HvKD19PbIePU175Q
1 changed files with 11 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
robots.txt
View File

@ -1,10 +1,21 @@
User-agent: *
Allow: /
Allow: /$
Allow: /blog/$
Allow: /blog/feed.xml
Allow: /blog/feed.en.xml
Allow: /blog/feed.fi.xml
Allow: /irc/$
Allow: /matrix/$
Allow: /n/$
Allow: /sitemap.xml
Allow: /sitemaps.xml
Disallow: /*.asc$
Disallow: /*.json$
Disallow: /*.sig$
Disallow: /*.txt$
Disallow: /*.toml$
Disallow: /*.xml$
Disallow: /txt/
Disallow: /php/
Disallow: /r/