Mercurial > libervia-web

changeset 952:0f309eeef26d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added robots.txt: robots.txt is added currently in src/browser/public/robots.txt because that's what is served at root with current Pyjamas implementation, this should change when Pyjamas will be removed (for Libervia 0.8). Current version uses "Allow" which is not recognized by all crawlers, but major ones should do.
author Goffi <goffi@goffi.org>
date Thu, 22 Jun 2017 09:09:02 +0200
parents 43df2bf38787
children 5cdd77190a3b
files src/browser/public/robots.txt
diffstat 1 files changed, 7 insertions(+), 0 deletions(-) [+]
line wrap: on
line diff
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/src/browser/public/robots.txt	Thu Jun 22 09:09:02 2017 +0200
@@ -0,0 +1,7 @@
+User-agent: *
+Allow: /blog/
+Allow: /u/
+Disallow: /
+
+User-agent: Mediapartners-*
+Disallow: /