annotate src/browser/public/robots.txt @ 1016:fc1c913cc9d1

pages (blog_new, common/blog): various blog improvments: - item_id can now be specified in URL parsing, and a filter_keyword is used so future variants can be implemented (e.g. tags) - if item_id is specified, rsm is discarded and only the requested item is retrieved - new template data "items_http_uri" is filled, with links to HTTP versions of items - blog_new is now a simple redirection to common/blog as the later should be generic enough to handle all use cases.
author Goffi <goffi@goffi.org>
date Fri, 19 Jan 2018 18:09:39 +0100
parents 0f309eeef26d
children a54882348ecc
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
952
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
1 User-agent: *
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
2 Allow: /blog/
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
3 Allow: /u/
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
4 Disallow: /
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
5
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
6 User-agent: Mediapartners-*
0f309eeef26d Added robots.txt:
Goffi <goffi@goffi.org>
parents:
diff changeset
7 Disallow: /