controlled allowing of letting the bots in

2024-05-31 06:29:21 +03:00 · 2024-05-31 06:29:21 +03:00 · 9bda780c40
parent 8ab675d87d
commit 9bda780c40
33 changed files with 76 additions and 47 deletions
--- a/_config.yml
+++ b/_config.yml
@ -12,7 +12,8 @@ baseurl: "" # the subpath of your site, e.g. /blog/
 # At least it's just a static page.
 #url: "https://www.aminda.eu" # the base hostname & protocol for your site
 url: "https://aminda.eu" # the base hostname & protocol for your site
-# Default meta-robots tags, can be set per-page
+# Default meta-robots tags, can be set per-page. SET PER PAGE, DON'T ALLOW
+# THEM HERE! Remember /n and less good blog posts etc.
 robots: noai, nofollow, noindex

 codeberg_username: Mikaela
--- a/blog/_posts/2015-03-25-leaving-bots-life.md
+++ b/blog/_posts/2015-03-25-leaving-bots-life.md
@ -5,6 +5,7 @@ title: "Little on my life currently"
 category: [english]
 tags: [irc, english, life]
 redirect_from: /english/2015/03/25/leaving-bots-life.html
+robots: noai
 ---

 ## A little on my life currently
--- a/blog/_posts/2015-05-31-weechat-filter.md
+++ b/blog/_posts/2015-05-31-weechat-filter.md
@ -8,6 +8,7 @@ redirect_from:
  - /filter.html
  - /weechat-filter.html
  - /english/2015/05/31/weechat-filter.html
+robots: noai
 ---

 _Everyone knows the `/ignore` command, but there is also `/filter` which
--- a/blog/_posts/2015-06-12-ufw.md
+++ b/blog/_posts/2015-06-12-ufw.md
@ -7,6 +7,7 @@ tags: [english]
 redirect_from:
  - /ufw/
  - /english/2015/06/12/ufw.html
+robots: noai
 ---

 _This post describes my UFW config and is here so I find it from somewhere
--- a/blog/_posts/2015-06-22-ipv6.md
+++ b/blog/_posts/2015-06-22-ipv6.md
@ -9,6 +9,7 @@ redirect_from:
  - /IPv6/
  - /english/2015/06/22/ipv6.html
 lang: en
+robots: noai
 ---

 _There appears to be a lot of confusion on IPv6 and in this post I try to
--- a/blog/_posts/2015-11-21-sahkopostia-lakivaliokunnalle-translaki.md
+++ b/blog/_posts/2015-11-21-sahkopostia-lakivaliokunnalle-translaki.md
@ -6,6 +6,7 @@ category: [finnish]
 tags: [finnish, translaki]
 redirect_from: /finnish/2015/11/21/sahkopostia-lakivaliokunnalle-translaki.html
 lang: fi
+robots: noai
 ---

 _Toivottavasti tämä viesti on edes jotenkuten järkevä, se olisi ollut
--- a/blog/_posts/2015-11-29-SSID_optout_nomap.md
+++ b/blog/_posts/2015-11-29-SSID_optout_nomap.md
@ -10,6 +10,7 @@ redirect_from:
  - /openwireless_nomap.html
  - /openwireless5_nomap.html
  - /english/2015/11/29/SSID_optout_nomap.html
+robots: noai
 ---

 _Anxiety is now focusing to WLAN mapping & key sharing and how easy it is
--- a/blog/_posts/2015-12-25-asus-tunnelbroker-net.md
+++ b/blog/_posts/2015-12-25-asus-tunnelbroker-net.md
@ -6,6 +6,7 @@ category: [english]
 tags: [IPv6, english]
 redirect_from: /english/2015/12/25/asus-tunnelbroker-net.html
 lang: en
+robots: noai
 ---

 _I am typing this, because there is a lot of misinformation on this,
--- a/blog/_posts/2017-04-30-post-trans.md
+++ b/blog/_posts/2017-04-30-post-trans.md
@ -5,6 +5,7 @@ title: "Post trans process, orchiectomy"
 category: [english]
 tags: [english, life, buddhism]
 redirect_from: /english/2017/04/30/post-trans.html
+robots: noai
 ---

 _A little confusing flood of what my fingers brought up on the last past
--- a/blog/_posts/2019-07-11-android-private-dns-in-practice.md
+++ b/blog/_posts/2019-07-11-android-private-dns-in-practice.md
@ -8,6 +8,7 @@ redirect_from:
  - /dns.html
  - /dot.html
 lang: en
+robots: noai
 ---

 _Since I first heard of Android 9 Private DNS I wondered how it will work
--- a/blog/_posts/2021-08-03-matrix-perfect-privacy-not.md
+++ b/blog/_posts/2021-08-03-matrix-perfect-privacy-not.md
@ -7,6 +7,7 @@ redirect_from:
  - /matrix.html
  - /element.html
 lang: en
+robots: noai
 ---

 _Having used Matrix since 2016 and hearing about its greatness without any
--- a/blog/_posts/2021-11-23-leaving-privacyguides.md
+++ b/blog/_posts/2021-11-23-leaving-privacyguides.md
@ -4,6 +4,7 @@ title: "Leaving Privacy Guides team"
 category: [english]
 tags: [privacyguides]
 lang: en
+robots: noai
 ---

 _PrivacyGuides.org was supposed to be the continuation of PrivacyTools.io
--- a/blog/_posts/2021-12-05-matrix-community-abuse-security-by-obscurity.md
+++ b/blog/_posts/2021-12-05-matrix-community-abuse-security-by-obscurity.md
@ -4,6 +4,7 @@ title: "Matrix abuse protection model for community maintainers: security by obs
 category: [english]
 tags: [matrix]
 lang: en
+robots: noai
 ---

 _I am administrator or moderator in multiple communities in Matrix, the most sizable
--- a/blog/_posts/2022-08-22-ssh-signing-verifying-functions.md
+++ b/blog/_posts/2022-08-22-ssh-signing-verifying-functions.md
@ -3,6 +3,7 @@ layout: post
 title: "bash/zsh functions for easier SSH signing and verification"
 category: [english]
 tags: [ssh]
+robots: noai
 ---

 _I have been using SSH signed git commits from 8 months and started signing things with my SSH key instead of PGP keys and thought to share how to do that more easily_
--- a/blog/_posts/2024-05-17-https-everywhere.md
+++ b/blog/_posts/2024-05-17-https-everywhere.md
@ -8,6 +8,7 @@ redirect_from:
  - /httpseverywhere.html
  - /https.html
 lang: en
+robots: noai
 ---

 _I used to be sad since the EFF discontinued HTTPS Everywhere extension since the setting often didn't sync and it only applied to me as opposed to everyone using a shared computer. However since I have dived into browser policies, this is no longer an issue for me._
--- a/blog/_posts/2024-05-22-policy-contentblocker.md
+++ b/blog/_posts/2024-05-22-policy-contentblocker.md
@ -8,6 +8,7 @@ redirect_from:
  - /ublockorigin.html
  - /https2.html
 lang: en
+robots: noai
 ---

 _I previously wrote about enforcing HTTPS for all users/profiles through browser policy receiving some positive feedback and I felt like continuing on the subject by instructing with extension installation. This barely scratches the surface of what browser policy can do for you either though._
--- a/blog/index.md
+++ b/blog/index.md
@ -5,6 +5,7 @@ navigation: true
 permalink: /blog/
 excerpt: "Blog index, posts in English and posts in Finnish — Blogin etusivu, postaukset englanniksi ja postaukset suomeksi."
 lang: en
+robots: noai
 ---

 <!-- editorconfig-checker-disable -->
--- a/index.html
+++ b/index.html
@ -2,6 +2,7 @@
 layout: index
 title: Index
 excerpt: "I am a Highly Sensitive Autistic Pirate with Linux experience since 2008. I am familiar with git and looking for employment."
+robots: noai
 ---

 <p id="avatar">
--- a/n/coauthor.md
+++ b/n/coauthor.md
@ -1,8 +1,9 @@
 ---
 layout: mini
 permalink: /n/coauthor.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---

 # Git forge private emails
--- a/n/dns.md
+++ b/n/dns.md
@ -6,8 +6,9 @@ redirect_from:
  - /n/rethink.html
  - /n/esni.html
  - /n/ech.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---

 # DNS notes
--- a/n/essentialsoftware.md
+++ b/n/essentialsoftware.md
@ -1,8 +1,9 @@
 ---
 layout: mini
 permalink: /n/essentialsoftware.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---

 # Essential software for new systems
--- a/n/f-droid.md
+++ b/n/f-droid.md
@ -2,8 +2,9 @@
 layout: mini
 permalink: /n/f-droid.html
 redirect_from: /n/fdroid.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---

 F-Droid is kind of apt for Android with multiple repositories available. This
--- a/n/fairbuds.md
+++ b/n/fairbuds.md
@ -3,8 +3,9 @@ layout: mini
 permalink: /n/fairbuds.html
 redirect_from:
  - /n/fairbudsxl.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---

 # Fairbuds XL configuration
--- a/n/firewalld.md
+++ b/n/firewalld.md
@ -3,8 +3,9 @@ layout: mini
 permalink: /n/firewalld.html
 redirect_from:
  - /n/firewall-cmd.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---

 # Quick note on firewalld usage
--- a/n/gpg.md
+++ b/n/gpg.md
@ -13,7 +13,8 @@ redirect_from:
  - /n/wkd.html
  - n/keybase.html
 lang: en
-sitemap: false
+sitemap: true
+robots: noai
 ---

 # Quick GPG notes where I may find them
--- a/n/keyoxide.md
+++ b/n/keyoxide.md
@ -2,8 +2,9 @@
 layout: mini
 permalink: /n/keyoxide.html
 redirect_from: /n/keyoxide-flutter.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---

 > Keyoxide is a decentralized tool to create and verify decentralized online identities.
--- a/n/localectl.md
+++ b/n/localectl.md
@ -6,8 +6,9 @@ redirect_from:
  - /locales.html
  - /english/2016/10/16/adjusting-locales.html
  - /blog/english/2016/10/16/adjusting-locales.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---

 # Quick notes on `localectl` settings
--- a/n/matrixspoilers.md
+++ b/n/matrixspoilers.md
@ -4,8 +4,9 @@ permalink: /n/matrixspoilers.html
 redirect_from:
  - n/nheko.html
  - n/matrixspoiler.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---

 The Matrix specification supports spoilers since version 1.1:
--- a/n/obtainium.md
+++ b/n/obtainium.md
@ -1,8 +1,9 @@
 ---
 layout: mini
 permalink: /n/obtainium.html
-sitemap: false
+sitemap: true
 lang: en
+robots: noai
 ---

 [Obtainium](https://github.com/ImranR98/Obtainium/) downloads apps directly from forge/whatever without need for app
--- a/pages/discuss.markdown
+++ b/pages/discuss.markdown
@ -6,6 +6,7 @@ redirect_from: /discuss/index.html
 excerpt: "Introduction to my public instant messaging chats"
 navigation: true
 lang: en
+robots: noai
 ---

 I have multiple instant messaging chatrooms.
--- a/pages/irc/irc.markdown
+++ b/pages/irc/irc.markdown
@ -4,6 +4,7 @@ title: IRC
 navigation: true
 permalink: /irc/
 excerpt: "Where do you find me at IRC and verify that it's me. Also includes my IRC related posts that are hopefully helpful."
+robots: noai
 ---

 IRC has been a big part of my life, I discovered it during junior high school
--- a/pages/matrix.markdown
+++ b/pages/matrix.markdown
@ -7,6 +7,7 @@ excerpt: "Matrix, a decentralised conversation store"
 redirect_from:
  - /matrix.html
 lang: en
+robots: noai
 ---

 Just like [IRC](/irc/), _Matrix_ has became a part of my social life online. My
--- a/robots.txt
+++ b/robots.txt
@ -1,48 +1,15 @@
 User-agent: *
 Allow: /
-Allow: /$
-Allow: /index.html
-Disallow: /blog/english/2015/
-Disallow: /blog/english/2016/
-Disallow: /blog/finnish/2015/
-Disallow: /blog/finnish/2016/
-Disallow: /blog/finnish/2017/
 Disallow: /txt/
 Disallow: /php/
 Disallow: /r/
 Disallow: /or/
-Disallow: /n/
-Noindex: /blog/english/2015/
-Noindex: /blog/english/2016/
-Noindex: /blog/finnish/2015/
-Noindex: /blog/finnish/2016/
-Noindex: /blog/finnish/2017/
-Noindex: /txt/
-Noindex: /php/
-Noindex: /r/
-Noindex: /or/
-Noindex: /n/
+Disallow: /ir/
 Sitemap: https://aminda.eu/sitemap.xml
 Sitemap: https://aminda.eu/sitemaps.xml

-# These were previously part of darkvisitors but are not anymore for some reason?
-User-agent: ChatGPT-User
-Disallow: /
-
-User-agent: cohere-ai
-Disallow: /
-
-User-agent: omgilibot
-Disallow: /
-
 # https://darkvisitors.com/robots-txt-builder
-# Dark Visitors robots.txt
-
-# AI Data Scraper
-# https://darkvisitors.com/agents/anthropic-ai
-
-User-agent: anthropic-ai
-Disallow: /
+# Dark Visitors Robots.txt

 # AI Data Scraper
 # https://darkvisitors.com/agents/bytespider
@ -56,6 +23,18 @@ Disallow: /
 User-agent: CCBot
 Disallow: /

+# AI Data Scraper
+# https://darkvisitors.com/agents/claudebot
+
+User-agent: ClaudeBot
+Disallow: /
+
+# AI Data Scraper
+# https://darkvisitors.com/agents/diffbot
+
+User-agent: Diffbot
+Disallow: /
+
 # AI Data Scraper
 # https://darkvisitors.com/agents/facebookbot

@ -79,3 +58,21 @@ Disallow: /

 User-agent: omgili
 Disallow: /
+
+# Undocumented AI Agent
+# https://darkvisitors.com/agents/anthropic-ai
+
+User-agent: anthropic-ai
+Disallow: /
+
+# Undocumented AI Agent
+# https://darkvisitors.com/agents/claude-web
+
+User-agent: Claude-Web
+Disallow: /
+
+# Undocumented AI Agent
+# https://darkvisitors.com/agents/cohere-ai
+
+User-agent: cohere-ai
+Disallow: /