|
5df02aed87
|
robots.txt: remove empty lines
Apparently some robots (or at least robots.txt validators) don't like
them.
|
2015-04-26 20:30:57 +03:00 |
|
|
083bfcc98a
|
robots.txt: be more strict
|
2015-04-26 20:24:28 +03:00 |
|
|
354683729f
|
fix robots.txt
I somehow misremembered that you Allow and Deny while in reality you
Allow and Disallow.
|
2015-04-26 20:10:47 +03:00 |
|
|
aec43c7c17
|
Does the sitemap really need to include 404 error?
|
2015-04-26 18:50:25 +03:00 |
|
|
0a2a8fb126
|
block bot & channel in sitemap.xml & robots.txt
|
2015-04-26 18:45:04 +03:00 |
|
|
c028d2d0de
|
fix 82f87bb41f
|
2015-02-06 08:46:58 +02:00 |
|
|
82f87bb41f
|
add robots.txt
|
2015-02-06 08:39:24 +02:00 |
|