Mercurial > libervia-web

annotate src/browser/public/robots.txt @ 1018:78af5457d3f8

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Pages: added url_cache setting: when url_cache is set, result of parse_url is cached (per profile, because in some pages date retrieved may differ between profiles), and then reused. This is useful if some data are fetched during parse_url (e.g. bridge/XMPP call).
author Goffi <goffi@goffi.org>
date Sun, 21 Jan 2018 13:08:54 +0100
parents 0f309eeef26d
children a54882348ecc
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
952
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
1 User-agent: *
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
2 Allow: /blog/
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
3 Allow: /u/
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
4 Disallow: /
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
5
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
6 User-agent: Mediapartners-*
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
7 Disallow: /