Mercurial > libervia-web

annotate src/browser/public/robots.txt @ 952:0f309eeef26d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added robots.txt: robots.txt is added currently in src/browser/public/robots.txt because that's what is served at root with current Pyjamas implementation, this should change when Pyjamas will be removed (for Libervia 0.8). Current version uses "Allow" which is not recognized by all crawlers, but major ones should do.
author Goffi <goffi@goffi.org>
date Thu, 22 Jun 2017 09:09:02 +0200
parents
children a54882348ecc
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
952
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
1 User-agent: *
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
2 Allow: /blog/
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
3 Allow: /u/
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
4 Disallow: /
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
5
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
6 User-agent: Mediapartners-*
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
7 Disallow: /