d340409002
robots.txt: Disallow /txt/ and /php/ as they don't contain HTML noindex
2020-11-23 13:39:27 +02:00
895a6b6903
robots.txt: remove Disallow so noindex can be read
2020-11-23 13:28:35 +02:00
9a43fe0364
robots.txt: allow /$ & /index.html
2019-12-08 16:48:38 +02:00
c90a96bd8a
robots.txt: opt out of search engines for now
2019-06-03 21:57:03 +03:00
b463dbbacc
add ir/ for I2P redirects
2019-05-06 12:49:51 +03:00
73949c0b60
robots.txt: Disallow & Deny reflinks as search engines are unhappy
2019-03-09 14:34:48 +02:00
294c14c91b
robots.txt: add Noindex
2019-02-22 19:18:10 +02:00
9f445cc622
or: add some onion not-redirects
2018-09-16 15:14:29 +03:00
8f817b5abd
[WIP] add pages/irc/znc just in case
2015-08-25 10:24:06 +03:00
0cf8c1c055
update roots.txt
...
[CI SKIP]
2015-08-25 10:12:30 +03:00
bc8110160a
Remove Tumblr
...
It doesn't like HSTS with included subdomains and enabling CloudFlare
would make CNAME check to domains.tumblr.com to fail.
sitemaps.xml: link to ciblia.tumblr.com sitemap directly.
2015-07-22 13:04:59 +03:00
8c4c8d10e5
pages/links2: hidden linklist on minorities
...
Closes #50
2015-06-12 19:58:09 +03:00
8e940ff288
add sitemaps.xml && update robots.txt & README
2015-05-02 10:34:18 +03:00
771c667c2a
r: redirects elsewhere
2015-05-01 10:23:02 +03:00
74951a8e88
robots.txt: add Tumblr sitemap
...
[SKIP CI]
2015-04-29 08:26:53 +03:00
5df02aed87
robots.txt: remove empty lines
...
Apparently some robots (or at least robots.txt validators) don't like
them.
2015-04-26 20:30:57 +03:00
083bfcc98a
robots.txt: be more strict
2015-04-26 20:24:28 +03:00
354683729f
fix robots.txt
...
I somehow misremembered that you Allow and Deny while in reality you
Allow and Disallow.
2015-04-26 20:10:47 +03:00
aec43c7c17
Does the sitemap really need to include 404 error?
2015-04-26 18:50:25 +03:00
0a2a8fb126
block bot & channel in sitemap.xml & robots.txt
2015-04-26 18:45:04 +03:00
c028d2d0de
fix 82f87bb41f
2015-02-06 08:46:58 +02:00
82f87bb41f
add robots.txt
2015-02-06 08:39:24 +02:00
