35 Commits

Author	SHA1	Message	Date
Aminda Suomalainen	40924ab27b	robots: nofollow, use the sitemap instead	2024-06-02 13:24:08 +03:00
Aminda Suomalainen	2798fca073	robots.txt: even more specific rules	2024-06-02 12:53:50 +03:00
Aminda Suomalainen	b23e622fae	robots.txt: add more explicit allow rules while disallowing xml	2024-06-02 12:35:16 +03:00
Aminda Suomalainen	b87a66d59a	robots.txt: disallow *.{asc,json,sig,txt,toml}	2024-06-02 12:17:35 +03:00
Aminda Suomalainen	9bda780c40	controlled allowing of letting the bots in	2024-05-31 06:30:07 +03:00
Aminda Suomalainen	ef09e6ce39	robots.txt: update from darkvisitors	2024-01-29 10:11:21 +02:00
Aminda Suomalainen	2e70384d1f	robots.txt: replace codeberg.org/robots.txt copying with Dark Visitors	2023-12-29 18:01:48 +02:00
Aminda Suomalainen	a7d4d4a453	robots.txt: block more AIs ... Copied from https://codeberg.org/robots.txt	2023-12-14 20:40:49 +02:00
Aminda Suomalainen	8c86d2b1b8	robots.txt: also disallow ChatGPT plugins	2023-12-14 20:39:57 +02:00
Aminda Suomalainen	6254a683fa	robots.txt: disallow GPTBot	2023-09-02 10:48:25 +03:00
Aminda Suomalainen	9405a4910f	more apex domain changes	2023-03-05 12:25:56 +02:00
Aminda Suomalainen	4e97e1679c	robots.txt: disallow/noindex blog posts pre 2018, r/, or/ & n/ ... Yes, I know Noindex is deprecated	2022-11-10 12:20:08 +02:00
Aminda Suomalainen	fe97829306	transfer domains	2022-03-06 14:37:23 +02:00
Mikaela Suomalainen	d340409002	robots.txt: Disallow /txt/ and /php/ as they don't contain HTML noindex	2020-11-23 13:39:27 +02:00
Mikaela Suomalainen	895a6b6903	robots.txt: remove Disallow so noindex can be read	2020-11-23 13:28:35 +02:00
Mikaela Suomalainen	9a43fe0364	robots.txt: allow /$ & /index.html	2019-12-08 16:48:38 +02:00
Mikaela Suomalainen	c90a96bd8a	robots.txt: opt out of search engines for now	2019-06-03 21:57:03 +03:00
Mikaela Suomalainen	b463dbbacc	add ir/ for I2P redirects	2019-05-06 12:49:51 +03:00
Mikaela Suomalainen	73949c0b60	robots.txt: Disallow & Deny reflinks as search engines are unhappy	2019-03-09 14:34:48 +02:00
Mikaela Suomalainen	294c14c91b	robots.txt: add Noindex	2019-02-22 19:18:10 +02:00
Mikaela Suomalainen	9f445cc622	or: add some onion not-redirects	2018-09-16 15:14:29 +03:00
Mikaela Suomalainen	8f817b5abd	[WIP] add pages/irc/znc just in case	2015-08-25 10:24:06 +03:00
Mikaela Suomalainen	0cf8c1c055	update roots.txt ... [CI SKIP]	2015-08-25 10:12:30 +03:00
Mikaela Suomalainen	bc8110160a	Remove Tumblr ... It doesn't like HSTS with included subdomains and enabling CloudFlare would make CNAME check to domains.tumblr.com to fail. sitemaps.xml: link to ciblia.tumblr.com sitemap directly.	2015-07-22 13:04:59 +03:00
Mikaela Suomalainen	8c4c8d10e5	pages/links2: hidden linklist on minorities ... Closes #50	2015-06-12 19:58:09 +03:00
Mikaela Suomalainen	8e940ff288	add sitemaps.xml && update robots.txt & README	2015-05-02 10:34:18 +03:00
Mikaela Suomalainen	771c667c2a	r: redirects elsewhere	2015-05-01 10:23:02 +03:00
Mikaela Suomalainen	74951a8e88	robots.txt: add Tumblr sitemap ... [SKIP CI]	2015-04-29 08:26:53 +03:00
Mikaela Suomalainen	5df02aed87	robots.txt: remove empty lines ... Apparently some robots (or at least robots.txt validators) don't like them.	2015-04-26 20:30:57 +03:00
Mikaela Suomalainen	083bfcc98a	robots.txt: be more strict	2015-04-26 20:24:28 +03:00
Mikaela Suomalainen	354683729f	fix robots.txt ... I somehow misremembered that you Allow and Deny while in reality you Allow and Disallow.	2015-04-26 20:10:47 +03:00
Mikaela Suomalainen	aec43c7c17	Does the sitemap really need to include 404 error?	2015-04-26 18:50:25 +03:00
Mikaela Suomalainen	0a2a8fb126	block bot & channel in sitemap.xml & robots.txt	2015-04-26 18:45:04 +03:00
Mikaela Suomalainen	c028d2d0de	fix 82f87bb41f	2015-02-06 08:46:58 +02:00
Mikaela Suomalainen	82f87bb41f	add robots.txt	2015-02-06 08:39:24 +02:00