Disallowing /wp-includes, /wp-content/plugins (cache and themes) affects my SEO score?
I want to have a clear SEO analyze for my website, and I want to be sure that disallowing some WP folders doesn’t affect my SEO.
Should I keep the robots.txt file simple as possible or to add some disallow attributes to keep the crawling tidy?
Thank you!
3
Answers
Yes, Disallowing /wp-includes, /wp-content/plugins (cache and themes) affects your SEO score. because search engines are not able to crawl your theme’s code and plugin’s code. Search engines are not able to know the site’s functionality. so I suggest not disallow these folders in robots.txt
Don’t mess with robots.txt unless you know exactly what’s going on.
The best, indeed the only reliable, way to present your WordPress site to search engine crawlers is to use an SEO plugin that generates a sitemap. That sitemap will contain useful information in the format the crawlers can use. This is especially true if you follow your chosen plugin’s suggestions about annotating your posts and pages.
Those plugins will handle robots.txt for you.
And, there’s no need to exclude WordPress code directories: unless your site’s security is dangerously misconfigured crawlers cannot see those directories anyway.
There’s nothing in
wp-includesorwp-content/pluginsorwp-content/themesthat search engines will be interested in, so while you probably can exclude them inrobots.txt, there’s really no reason to, unless you’ve been hacked or have malware.