Dave Jarvis' Repositories

git clone https://repo.autonoma.ca/repo/keenwrite.git
keenwrite / Commits / 37ba450

Disallows certain robots from crawling repository

AuthorDaveJarvis <email>
Date2023-11-07 18:45:06 GMT-0800
Commit37ba4502e3c5d7d6723fac0b8f0595053166cdf3
Parentbf0f683
Delta10 lines added, 0 lines removed, 10-line increase
www/robots.txt
crawl-delay: 60
+user-agent: Googlebot
+disallow: /repository/*
+user-agent: Bingbot
+disallow: /repository/*
+user-agent: YandexBot
+disallow: /repository/*
+user-agent: MicrosoftBot
+disallow: /repository/*
+
+