mikaela.github.io/robots.txt

User-agent: *
Allow: /
Disallow: /404
Disallow: /bot
Disallow: /channel
Disallow: /links2
Disallow: /irc/bot
Disallow: /irc/channel
Disallow: /irc/znc
Disallow: /r/
Sitemap: https://mikaela.info/sitemap.xml
Sitemap: https://mikaela.info/sitemaps.xml
add robots.txt 2015-02-06 07:39:24 +01:00			`User-agent: *`
			`Allow: /`
robots.txt: be more strict 2015-04-26 19:24:28 +02:00			`Disallow: /404`
			`Disallow: /bot`
			`Disallow: /channel`
pages/links2: hidden linklist on minorities Closes #50 2015-06-12 18:58:09 +02:00			`Disallow: /links2`
update roots.txt [CI SKIP] 2015-08-25 09:12:30 +02:00			`Disallow: /irc/bot`
			`Disallow: /irc/channel`
[WIP] add pages/irc/znc just in case 2015-08-25 09:24:06 +02:00			`Disallow: /irc/znc`
r: redirects elsewhere 2015-05-01 09:23:02 +02:00			`Disallow: /r/`
fix 82f87bb41fae166f4a8a5ca670fd139b6fe5bca8 2015-02-06 07:46:58 +01:00			`Sitemap: https://mikaela.info/sitemap.xml`
add sitemaps.xml && update robots.txt & README 2015-05-02 09:34:18 +02:00			`Sitemap: https://mikaela.info/sitemaps.xml`