# HG changeset patch
# User Goffi <goffi@goffi.org>
# Date 1498115342 -7200
# Node ID 0f309eeef26d0c12fd4780f3ddc9cefbdc3cee17
# Parent  43df2bf38787fc2fe8496ef68c1bbdb7ba24f3bc
Added robots.txt:

robots.txt is added currently in src/browser/public/robots.txt because that's what is served at root with current Pyjamas implementation, this should change when Pyjamas will be removed (for Libervia 0.8).
Current version uses "Allow" which is not recognized by all crawlers, but major ones should do.

diff -r 43df2bf38787 -r 0f309eeef26d src/browser/public/robots.txt
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/src/browser/public/robots.txt	Thu Jun 22 09:09:02 2017 +0200
@@ -0,0 +1,7 @@
+User-agent: *
+Allow: /blog/
+Allow: /u/
+Disallow: /
+
+User-agent: Mediapartners-*
+Disallow: /