mikaela.github.io/robots.txt

User-agent: *
Allow: /
Disallow: /404
Noindex: /404
Disallow: /bot
Noindex: /bot
Disallow: /channel
Noindex: /channel
Disallow: /links2
Noindex: /links2
Disallow: /irc/bot
Noindex: /irc/bot
Disallow: /irc/channel
Noindex: /irc/channel
Disallow: /irc/znc
Noindex: /irc/znc
Disallow: /r/
Noindex: /r/
Disallow: /or/
Noindex: /or/
Disallow: /reflinks
Noindex: /reflinks
Sitemap: https://mikaela.info/sitemap.xml
Sitemap: https://mikaela.info/sitemaps.xml
add robots.txt 2015-02-06 07:39:24 +01:00			`User-agent: *`
			`Allow: /`
robots.txt: be more strict 2015-04-26 19:24:28 +02:00			`Disallow: /404`
robots.txt: add Noindex 2019-02-22 18:18:10 +01:00			`Noindex: /404`
robots.txt: be more strict 2015-04-26 19:24:28 +02:00			`Disallow: /bot`
robots.txt: add Noindex 2019-02-22 18:18:10 +01:00			`Noindex: /bot`
robots.txt: be more strict 2015-04-26 19:24:28 +02:00			`Disallow: /channel`
robots.txt: add Noindex 2019-02-22 18:18:10 +01:00			`Noindex: /channel`
pages/links2: hidden linklist on minorities Closes #50 2015-06-12 18:58:09 +02:00			`Disallow: /links2`
robots.txt: add Noindex 2019-02-22 18:18:10 +01:00			`Noindex: /links2`
update roots.txt [CI SKIP] 2015-08-25 09:12:30 +02:00			`Disallow: /irc/bot`
robots.txt: add Noindex 2019-02-22 18:18:10 +01:00			`Noindex: /irc/bot`
update roots.txt [CI SKIP] 2015-08-25 09:12:30 +02:00			`Disallow: /irc/channel`
robots.txt: add Noindex 2019-02-22 18:18:10 +01:00			`Noindex: /irc/channel`
[WIP] add pages/irc/znc just in case 2015-08-25 09:24:06 +02:00			`Disallow: /irc/znc`
robots.txt: add Noindex 2019-02-22 18:18:10 +01:00			`Noindex: /irc/znc`
r: redirects elsewhere 2015-05-01 09:23:02 +02:00			`Disallow: /r/`
robots.txt: add Noindex 2019-02-22 18:18:10 +01:00			`Noindex: /r/`
or: add some onion not-redirects 2018-09-16 14:03:27 +02:00			`Disallow: /or/`
robots.txt: add Noindex 2019-02-22 18:18:10 +01:00			`Noindex: /or/`
robots.txt: Disallow & Deny reflinks as search engines are unhappy 2019-03-09 13:34:48 +01:00			`Disallow: /reflinks`
			`Noindex: /reflinks`
fix 82f87bb41fae166f4a8a5ca670fd139b6fe5bca8 2015-02-06 07:46:58 +01:00			`Sitemap: https://mikaela.info/sitemap.xml`
robots.txt: Disallow & Deny reflinks as search engines are unhappy 2019-03-09 13:34:48 +01:00			`Sitemap: https://mikaela.info/sitemaps.xml`