Commit Graph

29 Commits

Author SHA1 Message Date
2e70384d1f
robots.txt: replace codeberg.org/robots.txt copying with Dark Visitors 2023-12-29 18:01:48 +02:00
a7d4d4a453
robots.txt: block more AIs
Copied from https://codeberg.org/robots.txt
2023-12-14 20:40:49 +02:00
8c86d2b1b8
robots.txt: also disallow ChatGPT plugins 2023-12-14 20:39:57 +02:00
6254a683fa
robots.txt: disallow GPTBot 2023-09-02 10:48:25 +03:00
9405a4910f
more apex domain changes 2023-03-05 12:25:56 +02:00
4e97e1679c
robots.txt: disallow/noindex blog posts pre 2018, r/, or/ & n/
Yes, I know Noindex is deprecated
2022-11-10 12:20:08 +02:00
fe97829306
transfer domains 2022-03-06 14:37:23 +02:00
d340409002
robots.txt: Disallow /txt/ and /php/ as they don't contain HTML noindex 2020-11-23 13:39:27 +02:00
895a6b6903
robots.txt: remove Disallow so noindex can be read 2020-11-23 13:28:35 +02:00
9a43fe0364
robots.txt: allow /$ & /index.html 2019-12-08 16:48:38 +02:00
c90a96bd8a
robots.txt: opt out of search engines for now 2019-06-03 21:57:03 +03:00
b463dbbacc
add ir/ for I2P redirects 2019-05-06 12:49:51 +03:00
73949c0b60
robots.txt: Disallow & Deny reflinks as search engines are unhappy 2019-03-09 14:34:48 +02:00
294c14c91b
robots.txt: add Noindex 2019-02-22 19:18:10 +02:00
9f445cc622
or: add some onion not-redirects 2018-09-16 15:14:29 +03:00
8f817b5abd [WIP] add pages/irc/znc just in case 2015-08-25 10:24:06 +03:00
0cf8c1c055 update roots.txt
[CI SKIP]
2015-08-25 10:12:30 +03:00
bc8110160a Remove Tumblr
It doesn't like HSTS with included subdomains and enabling CloudFlare
would make CNAME check to domains.tumblr.com to fail.

sitemaps.xml: link to ciblia.tumblr.com sitemap directly.
2015-07-22 13:04:59 +03:00
8c4c8d10e5 pages/links2: hidden linklist on minorities
Closes #50
2015-06-12 19:58:09 +03:00
8e940ff288 add sitemaps.xml && update robots.txt & README 2015-05-02 10:34:18 +03:00
771c667c2a r: redirects elsewhere 2015-05-01 10:23:02 +03:00
74951a8e88 robots.txt: add Tumblr sitemap
[SKIP CI]
2015-04-29 08:26:53 +03:00
5df02aed87 robots.txt: remove empty lines
Apparently some robots (or at least robots.txt validators) don't like
them.
2015-04-26 20:30:57 +03:00
083bfcc98a robots.txt: be more strict 2015-04-26 20:24:28 +03:00
354683729f fix robots.txt
I somehow misremembered that you Allow and Deny while in reality you
Allow and Disallow.
2015-04-26 20:10:47 +03:00
aec43c7c17 Does the sitemap really need to include 404 error? 2015-04-26 18:50:25 +03:00
0a2a8fb126 block bot & channel in sitemap.xml & robots.txt 2015-04-26 18:45:04 +03:00
c028d2d0de fix 82f87bb41f 2015-02-06 08:46:58 +02:00
82f87bb41f add robots.txt 2015-02-06 08:39:24 +02:00