- Betroffene Version
- WoltLab Suite 3.0
hi
Which is the Best robots.txt file for my website
Animon09
hi
Which is the Best robots.txt file for my website
Animon09
Should there be instructions in the Robots.txt file to block the calendar and the ACP?
I've read that Google spends ages crawling calendar pages that use up unnecessary resources.
Many thanks
Jupiter
Should there be instructions in the Robots.txt file to block the calendar and the ACP?
I don't know why you want to block the calendar, but the ACP doesn't make so much advantage, because google won't index it, because it has a meta tag set, that basically says that crawlers should not index the page.
Thanks Jens. I'll stick to what WoltLab uses for its robots.txt file as you helpfully illustrated.
Added that line too!
Appreciated!
Any folders like Cache that has an htaccess file in blocking all access to the folder, you don't need to add those folders to a robots.txt file because bots are already blocked from any that contain one. And WBB has it pretty well covered with an htaccess file added in a few folders blocking bots they (or anyone else) shouldn't be browsing. So there's not much need adding much of anything to a robots.txt file apart from pointing Google to the Sitemap file.
I'm surprised WBB doesn't now include a robots.txt file with 3.1 doing that actually, pointing google to the sitemap location.
I get that error now in Google webmaster
You simply don’t need that line.
Yes, i have remove it for now.
Thx.
I get that error now in Google webmaster
Hmm... might remove that myself then now if gets flagged up in Google webmaster
Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!