Aminda Suomalainen
b87a66d59a
robots.txt: disallow *.{asc,json,sig,txt,toml}
2024-06-02 12:17:35 +03:00
Aminda Suomalainen
9bda780c40
controlled allowing of letting the bots in
2024-05-31 06:30:07 +03:00
Aminda Suomalainen
ef09e6ce39
robots.txt: update from darkvisitors
2024-01-29 10:11:21 +02:00
Aminda Suomalainen
2e70384d1f
robots.txt: replace codeberg.org/robots.txt copying with Dark Visitors
2023-12-29 18:01:48 +02:00
Aminda Suomalainen
a7d4d4a453
robots.txt: block more AIs
...
Copied from https://codeberg.org/robots.txt
2023-12-14 20:40:49 +02:00
Aminda Suomalainen
8c86d2b1b8
robots.txt: also disallow ChatGPT plugins
2023-12-14 20:39:57 +02:00
Aminda Suomalainen
6254a683fa
robots.txt: disallow GPTBot
2023-09-02 10:48:25 +03:00
Aminda Suomalainen
9405a4910f
more apex domain changes
2023-03-05 12:25:56 +02:00
Aminda Suomalainen
4e97e1679c
robots.txt: disallow/noindex blog posts pre 2018, r/, or/ & n/
...
Yes, I know Noindex is deprecated
2022-11-10 12:20:08 +02:00
Aminda Suomalainen
fe97829306
transfer domains
2022-03-06 14:37:23 +02:00
Aminda Suomalainen
d340409002
robots.txt: Disallow /txt/ and /php/ as they don't contain HTML noindex
2020-11-23 13:39:27 +02:00
Aminda Suomalainen
895a6b6903
robots.txt: remove Disallow so noindex can be read
2020-11-23 13:28:35 +02:00
Aminda Suomalainen
9a43fe0364
robots.txt: allow /$ & /index.html
2019-12-08 16:48:38 +02:00
Aminda Suomalainen
c90a96bd8a
robots.txt: opt out of search engines for now
2019-06-03 21:57:03 +03:00
Aminda Suomalainen
b463dbbacc
add ir/ for I2P redirects
2019-05-06 12:49:51 +03:00
Aminda Suomalainen
73949c0b60
robots.txt: Disallow & Deny reflinks as search engines are unhappy
2019-03-09 14:34:48 +02:00
Aminda Suomalainen
294c14c91b
robots.txt: add Noindex
2019-02-22 19:18:10 +02:00
Aminda Suomalainen
9f445cc622
or: add some onion not-redirects
2018-09-16 15:14:29 +03:00
Aminda Suomalainen
8f817b5abd
[WIP] add pages/irc/znc just in case
2015-08-25 10:24:06 +03:00
Aminda Suomalainen
0cf8c1c055
update roots.txt
...
[CI SKIP]
2015-08-25 10:12:30 +03:00
Aminda Suomalainen
bc8110160a
Remove Tumblr
...
It doesn't like HSTS with included subdomains and enabling CloudFlare
would make CNAME check to domains.tumblr.com to fail.
sitemaps.xml: link to ciblia.tumblr.com sitemap directly.
2015-07-22 13:04:59 +03:00
Aminda Suomalainen
8c4c8d10e5
pages/links2: hidden linklist on minorities
...
Closes #50
2015-06-12 19:58:09 +03:00
Aminda Suomalainen
8e940ff288
add sitemaps.xml && update robots.txt & README
2015-05-02 10:34:18 +03:00
Aminda Suomalainen
771c667c2a
r: redirects elsewhere
2015-05-01 10:23:02 +03:00
Aminda Suomalainen
74951a8e88
robots.txt: add Tumblr sitemap
...
[SKIP CI]
2015-04-29 08:26:53 +03:00
Aminda Suomalainen
5df02aed87
robots.txt: remove empty lines
...
Apparently some robots (or at least robots.txt validators) don't like
them.
2015-04-26 20:30:57 +03:00
Aminda Suomalainen
083bfcc98a
robots.txt: be more strict
2015-04-26 20:24:28 +03:00
Aminda Suomalainen
354683729f
fix robots.txt
...
I somehow misremembered that you Allow and Deny while in reality you
Allow and Disallow.
2015-04-26 20:10:47 +03:00
Aminda Suomalainen
aec43c7c17
Does the sitemap really need to include 404 error?
2015-04-26 18:50:25 +03:00
Aminda Suomalainen
0a2a8fb126
block bot & channel in sitemap.xml & robots.txt
2015-04-26 18:45:04 +03:00
Aminda Suomalainen
c028d2d0de
fix 82f87bb41f
2015-02-06 08:46:58 +02:00
Aminda Suomalainen
82f87bb41f
add robots.txt
2015-02-06 08:39:24 +02:00