Mikaela/cv
1
0
Fork 0
mirror of https://github.com/Mikaela/cv.git synced 2025-08-17 18:17:22 +02:00
Code Releases Activity

Compare commits

base: Mikaela:1dca41a91a4ad9e6f57da0a90536a3b5cee5a18e
Branches Tags
Mikaela:gh-pages
Mikaela:pre-commit-ci-update-config
...
compare: Mikaela:6a19e95d03e948ee239e40a92bbc66798bdf23fd
Branches Tags
Mikaela:pre-commit-ci-update-config
Mikaela:gh-pages

2 Commits
1dca41a91a ... 6a19e95d03

Author SHA1 Message Date
Aminda Suomalainen
6a19e95d03
fix robots handling 2024-06-02 13:34:35 +03:00
Aminda Suomalainen
dde3294601
robots.txt: sync with aminda.eu 
while naturally removing irrelevant parts
 2024-06-02 12:58:30 +03:00
5 changed files with 77 additions and 3 deletions
Show Stats Expand all files Collapse all files

3
404.md
View File

@ -1,7 +1,8 @@
---
layout: page
permalink: /404.html
sitemap: true
robots: noindex, nofollow, noai
sitemap: false
excerpt: "404"
lang: fi
---

1
README.md
View File

@ -7,6 +7,7 @@ date: 2024-05-28T08:59:32+03:00
lang: en
permalink: /en/index.html
redirect_from: /en/html
robots: noai, nofollow
---
# Aminda Suomalainen

2
_includes/head.html
View File

@ -1,6 +1,6 @@
<head>
<meta charset="utf-8">
<meta name="robots" content="noai">
<meta name="robots" content="{{ page.robots }}">
<title>{{site.title}} {{ page.title }} — {{site.author}}</title>
<meta name="description" content="{% if page.excerpt %}{{ page.excerpt | strip_html | strip_newlines | truncate: 160 }}{% else %}{{ site.description | strip_html | strip_newlines }}{% endif %}">
<meta name="viewport" content="width=device-width, initial-scale=1">

1
fi/README.md
View File

@ -12,6 +12,7 @@ redirect_from:
- fi/index.html
- fi/README.html
- README.html
robots: nofollow, noai
---
# Aminda Suomalainen

73
robots.txt
View File

@ -1,6 +1,77 @@
User-agent: *
Allow: /
Allow: /$
Allow: /en/$
Allow: /*.html$
Allow: /robots.txt
Allow: /sitemap.xml
Disallow: /
Sitemap: https://cv.aminda.eu/sitemap.xml
# https://darkvisitors.com/robots-txt-builder
# Dark Visitors Robots.txt
# AI Data Scraper
# https://darkvisitors.com/agents/bytespider
User-agent: Bytespider
Disallow: /
# AI Data Scraper
# https://darkvisitors.com/agents/ccbot
User-agent: CCBot
Disallow: /
# AI Data Scraper
# https://darkvisitors.com/agents/claudebot
User-agent: ClaudeBot
Disallow: /
# AI Data Scraper
# https://darkvisitors.com/agents/diffbot
User-agent: Diffbot
Disallow: /
# AI Data Scraper
# https://darkvisitors.com/agents/facebookbot
User-agent: FacebookBot
Disallow: /
# AI Data Scraper
# https://darkvisitors.com/agents/google-extended
User-agent: Google-Extended
Disallow: /
# AI Data Scraper
# https://darkvisitors.com/agents/gptbot
User-agent: GPTBot
Disallow: /
# AI Data Scraper
# https://darkvisitors.com/agents/omgili
User-agent: omgili
Disallow: /
# Undocumented AI Agent
# https://darkvisitors.com/agents/anthropic-ai
User-agent: anthropic-ai
Disallow: /
# Undocumented AI Agent
# https://darkvisitors.com/agents/claude-web
User-agent: Claude-Web
Disallow: /
# Undocumented AI Agent
# https://darkvisitors.com/agents/cohere-ai
User-agent: cohere-ai
Disallow: /