Mercurial > libervia-web

view src/browser/public/robots.txt @ 952:0f309eeef26d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added robots.txt: robots.txt is added currently in src/browser/public/robots.txt because that's what is served at root with current Pyjamas implementation, this should change when Pyjamas will be removed (for Libervia 0.8). Current version uses "Allow" which is not recognized by all crawlers, but major ones should do.
author Goffi <goffi@goffi.org>
date Thu, 22 Jun 2017 09:09:02 +0200
parents
children a54882348ecc
line wrap: on
line source

User-agent: *
Allow: /blog/
Allow: /u/
Disallow: /

User-agent: Mediapartners-*
Disallow: /