summaryrefslogtreecommitdiffstats
path: root/robots.txt
AgeCommit message (Collapse)AuthorFilesLines
2013-06-17Bug 883013: googlebot is indexing more pages than it shouldByron Jones1-11/+12
2013-04-11Bug 860797 - add product dashboards to robots.txtDave Lawrence1-0/+1
2013-02-15Bug 839970 - sitemap extension needs to allow access to ↵Dave Lawrence1-0/+1
data/SiteMapIndex/sitemap* in robots.txt
2013-02-15Bug 839969 - Allow describekeywords.cgi in robots.txtDave Lawrence1-0/+1
2011-11-11Bug 700876 - robots.txt - discourage search engines from indexing duplicate ↵Dave Lawrence1-3/+4
information r=glob
2011-10-05more porting workDavid Lawrence1-2/+11
2002-04-25Bug 139800 - remove errant references to index.html. Patch by ddk, 2xr=gerv.gerv%gerv.net1-1/+1
2000-07-29If the application is set up at the server root (e.g. bugzilla.mozilla.org) ↵endico%mozilla.org1-0/+3
then this robots.txt disallows robots from scanning the site except for the top level file.