controlled allowing of letting the bots in

_config.yml

@@ -12,7 +12,8 @@ baseurl: "" # the subpath of your site, e.g. /blog/
 # At least it's just a static page.
 #url: "https://www.aminda.eu" # the base hostname & protocol for your site
 url: "https://aminda.eu" # the base hostname & protocol for your site
-# Default meta-robots tags, can be set per-page
+# Default meta-robots tags, can be set per-page. SET PER PAGE, DON'T ALLOW
+# THEM HERE! Remember /n and less good blog posts etc.
 robots: noai, nofollow, noindex
 
 codeberg_username: Mikaela

blog/_posts/2015-03-25-leaving-bots-life.md

@@ -5,6 +5,7 @@ title: "Little on my life currently"
 category: [english]
 tags: [irc, english, life]
 redirect_from: /english/2015/03/25/leaving-bots-life.html
+robots: noai
 ---
 
 ## A little on my life currently

blog/_posts/2015-05-31-weechat-filter.md

@@ -8,6 +8,7 @@ redirect_from:
   - /filter.html
   - /weechat-filter.html
   - /english/2015/05/31/weechat-filter.html
+robots: noai
 ---
 
 _Everyone knows the `/ignore` command, but there is also `/filter` which

blog/_posts/2015-06-12-ufw.md

@@ -7,6 +7,7 @@ tags: [english]
 redirect_from:
   - /ufw/
   - /english/2015/06/12/ufw.html
+robots: noai
 ---
 
 _This post describes my UFW config and is here so I find it from somewhere

blog/_posts/2015-06-22-ipv6.md

@@ -9,6 +9,7 @@ redirect_from:
   - /IPv6/
   - /english/2015/06/22/ipv6.html
 lang: en
+robots: noai
 ---
 
 _There appears to be a lot of confusion on IPv6 and in this post I try to

blog/_posts/2015-11-21-sahkopostia-lakivaliokunnalle-translaki.md

@@ -6,6 +6,7 @@ category: [finnish]
 tags: [finnish, translaki]
 redirect_from: /finnish/2015/11/21/sahkopostia-lakivaliokunnalle-translaki.html
 lang: fi
+robots: noai
 ---
 
 _Toivottavasti tämä viesti on edes jotenkuten järkevä, se olisi ollut

blog/_posts/2015-11-29-SSID_optout_nomap.md

@@ -10,6 +10,7 @@ redirect_from:
   - /openwireless_nomap.html
   - /openwireless5_nomap.html
   - /english/2015/11/29/SSID_optout_nomap.html
+robots: noai
 ---
 
 _Anxiety is now focusing to WLAN mapping & key sharing and how easy it is

blog/_posts/2015-12-25-asus-tunnelbroker-net.md

@@ -6,6 +6,7 @@ category: [english]
 tags: [IPv6, english]
 redirect_from: /english/2015/12/25/asus-tunnelbroker-net.html
 lang: en
+robots: noai
 ---
 
 _I am typing this, because there is a lot of misinformation on this,

blog/_posts/2017-04-30-post-trans.md

@@ -5,6 +5,7 @@ title: "Post trans process, orchiectomy"
 category: [english]
 tags: [english, life, buddhism]
 redirect_from: /english/2017/04/30/post-trans.html
+robots: noai
 ---
 
 _A little confusing flood of what my fingers brought up on the last past

blog/_posts/2019-07-11-android-private-dns-in-practice.md

@@ -8,6 +8,7 @@ redirect_from:
   - /dns.html
   - /dot.html
 lang: en
+robots: noai
 ---
 
 _Since I first heard of Android 9 Private DNS I wondered how it will work

blog/_posts/2021-08-03-matrix-perfect-privacy-not.md

@@ -7,6 +7,7 @@ redirect_from:
   - /matrix.html
   - /element.html
 lang: en
+robots: noai
 ---
 
 _Having used Matrix since 2016 and hearing about its greatness without any

blog/_posts/2021-11-23-leaving-privacyguides.md

@@ -4,6 +4,7 @@ title: "Leaving Privacy Guides team"
 category: [english]
 tags: [privacyguides]
 lang: en
+robots: noai
 ---
 
 _PrivacyGuides.org was supposed to be the continuation of PrivacyTools.io

blog/_posts/2021-12-05-matrix-community-abuse-security-by-obscurity.md

@@ -4,6 +4,7 @@ title: "Matrix abuse protection model for community maintainers: security by obs
 category: [english]
 tags: [matrix]
 lang: en
+robots: noai
 ---
 
 _I am administrator or moderator in multiple communities in Matrix, the most sizable

blog/_posts/2022-08-22-ssh-signing-verifying-functions.md

@@ -3,6 +3,7 @@ layout: post
 title: "bash/zsh functions for easier SSH signing and verification"
 category: [english]
 tags: [ssh]
+robots: noai
 ---
 
 _I have been using SSH signed git commits from 8 months and started signing things with my SSH key instead of PGP keys and thought to share how to do that more easily_

blog/_posts/2024-05-17-https-everywhere.md

@@ -8,6 +8,7 @@ redirect_from:
   - /httpseverywhere.html
   - /https.html
 lang: en
+robots: noai
 ---
 
 _I used to be sad since the EFF discontinued HTTPS Everywhere extension since the setting often didn't sync and it only applied to me as opposed to everyone using a shared computer. However since I have dived into browser policies, this is no longer an issue for me._

blog/_posts/2024-05-22-policy-contentblocker.md

@@ -8,6 +8,7 @@ redirect_from:
   - /ublockorigin.html
   - /https2.html
 lang: en
+robots: noai
 ---
 
 _I previously wrote about enforcing HTTPS for all users/profiles through browser policy receiving some positive feedback and I felt like continuing on the subject by instructing with extension installation. This barely scratches the surface of what browser policy can do for you either though._

blog/index.md

@@ -5,6 +5,7 @@ navigation: true
 permalink: /blog/
 excerpt: "Blog index, posts in English and posts in Finnish — Blogin etusivu, postaukset englanniksi ja postaukset suomeksi."
 lang: en
+robots: noai
 ---
 
 <!-- editorconfig-checker-disable -->

index.html

@@ -2,6 +2,7 @@
 layout: index
 title: Index
 excerpt: "I am a Highly Sensitive Autistic Pirate with Linux experience since 2008. I am familiar with git and looking for employment."
+robots: noai
 ---
 
 <p id="avatar">

n/coauthor.md

@@ -1,8 +1,9 @@
 ---
 layout: mini
 permalink: /n/coauthor.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---
 
 # Git forge private emails

n/dns.md

@@ -6,8 +6,9 @@ redirect_from:
   - /n/rethink.html
   - /n/esni.html
   - /n/ech.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---
 
 # DNS notes

n/essentialsoftware.md

@@ -1,8 +1,9 @@
 ---
 layout: mini
 permalink: /n/essentialsoftware.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---
 
 # Essential software for new systems

n/f-droid.md

@@ -2,8 +2,9 @@
 layout: mini
 permalink: /n/f-droid.html
 redirect_from: /n/fdroid.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---
 
 F-Droid is kind of apt for Android with multiple repositories available. This

n/fairbuds.md

@@ -3,8 +3,9 @@ layout: mini
 permalink: /n/fairbuds.html
 redirect_from:
   - /n/fairbudsxl.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---
 
 # Fairbuds XL configuration

n/firewalld.md

@@ -3,8 +3,9 @@ layout: mini
 permalink: /n/firewalld.html
 redirect_from:
   - /n/firewall-cmd.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---
 
 # Quick note on firewalld usage

n/gpg.md

@@ -13,7 +13,8 @@ redirect_from:
   - /n/wkd.html
   - n/keybase.html
 lang: en
-sitemap: false
+sitemap: true
+robots: noai
 ---
 
 # Quick GPG notes where I may find them

n/keyoxide.md

@@ -2,8 +2,9 @@
 layout: mini
 permalink: /n/keyoxide.html
 redirect_from: /n/keyoxide-flutter.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---
 
 > Keyoxide is a decentralized tool to create and verify decentralized online identities.

n/localectl.md

@@ -6,8 +6,9 @@ redirect_from:
   - /locales.html
   - /english/2016/10/16/adjusting-locales.html
   - /blog/english/2016/10/16/adjusting-locales.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---
 
 # Quick notes on `localectl` settings

n/matrixspoilers.md

@@ -4,8 +4,9 @@ permalink: /n/matrixspoilers.html
 redirect_from:
   - n/nheko.html
   - n/matrixspoiler.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---
 
 The Matrix specification supports spoilers since version 1.1:

n/obtainium.md

@@ -1,8 +1,9 @@
 ---
 layout: mini
 permalink: /n/obtainium.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---
 
 [Obtainium](https://github.com/ImranR98/Obtainium/) downloads apps directly from forge/whatever without need for app

pages/discuss.markdown

@@ -6,6 +6,7 @@ redirect_from: /discuss/index.html
 excerpt: "Introduction to my public instant messaging chats"
 navigation: true
 lang: en
+robots: noai
 ---
 
 I have multiple instant messaging chatrooms.

pages/irc/irc.markdown

@@ -4,6 +4,7 @@ title: IRC
 navigation: true
 permalink: /irc/
 excerpt: "Where do you find me at IRC and verify that it's me. Also includes my IRC related posts that are hopefully helpful."
+robots: noai
 ---
 
 IRC has been a big part of my life, I discovered it during junior high school

pages/matrix.markdown

@@ -7,6 +7,7 @@ excerpt: "Matrix, a decentralised conversation store"
 redirect_from:
   - /matrix.html
 lang: en
+robots: noai
 ---
 
 Just like [IRC](/irc/), _Matrix_ has became a part of my social life online. My

robots.txt

@@ -1,48 +1,15 @@
 User-agent: *
 Allow: /
-Allow: /$
-Allow: /index.html
-Disallow: /blog/english/2015/
-Disallow: /blog/english/2016/
-Disallow: /blog/finnish/2015/
-Disallow: /blog/finnish/2016/
-Disallow: /blog/finnish/2017/
 Disallow: /txt/
 Disallow: /php/
 Disallow: /r/
 Disallow: /or/
-Disallow: /n/
+Disallow: /ir/
-Noindex: /blog/english/2015/
-Noindex: /blog/english/2016/
-Noindex: /blog/finnish/2015/
-Noindex: /blog/finnish/2016/
-Noindex: /blog/finnish/2017/
-Noindex: /txt/
-Noindex: /php/
-Noindex: /r/
-Noindex: /or/
-Noindex: /n/
 Sitemap: https://aminda.eu/sitemap.xml
 Sitemap: https://aminda.eu/sitemaps.xml
 
-# These were previously part of darkvisitors but are not anymore for some reason?
-User-agent: ChatGPT-User
-Disallow: /
-
-User-agent: cohere-ai
-Disallow: /
-
-User-agent: omgilibot
-Disallow: /
-
 # https://darkvisitors.com/robots-txt-builder
-# Dark Visitors robots.txt
+# Dark Visitors Robots.txt
-
-# AI Data Scraper
-# https://darkvisitors.com/agents/anthropic-ai
-
-User-agent: anthropic-ai
-Disallow: /
 
 # AI Data Scraper
 # https://darkvisitors.com/agents/bytespider
@@ -56,6 +23,18 @@ Disallow: /
 User-agent: CCBot
 Disallow: /
 
+# AI Data Scraper
+# https://darkvisitors.com/agents/claudebot
+
+User-agent: ClaudeBot
+Disallow: /
+
+# AI Data Scraper
+# https://darkvisitors.com/agents/diffbot
+
+User-agent: Diffbot
+Disallow: /
+
 # AI Data Scraper
 # https://darkvisitors.com/agents/facebookbot
 
@@ -79,3 +58,21 @@ Disallow: /
 
 User-agent: omgili
 Disallow: /
+
+# Undocumented AI Agent
+# https://darkvisitors.com/agents/anthropic-ai
+
+User-agent: anthropic-ai
+Disallow: /
+
+# Undocumented AI Agent
+# https://darkvisitors.com/agents/claude-web
+
+User-agent: Claude-Web
+Disallow: /
+
+# Undocumented AI Agent
+# https://darkvisitors.com/agents/cohere-ai
+
+User-agent: cohere-ai
+Disallow: /