friends. I have a question regarding PrestaShop off-the-shelf robots.txt file.
Is it a normal practice to allow Search engines to index PrestaShop */module/*
route?
Can please somebody comment:
A) Does it works as expected? [ */module/*
indexation is necessary for the good SEO ]
, or
B) this is an issue – security breach, as Search engine is indexing PrestaShop internal technical structure and exposing it to the rest of the world?
Thank you.
2
Answers
Some of the modules use “/module/” route for their front pages So it is a good idea to allow this on robots.txt file
But I do not recommend allowing the “/modules/” route as this can cause security problems for you
For SEO you need show crawlers only pages you believe are useful for users and you want user to find those content.
It means that if under /module/ you have pages with content you want user to see from search engine – you may open it in robots and make sure that there is no noindex tag in pages HTML.
Also note if you put /module/ to disallow, but you have inbound external links to pages in /module/, those pages could be indexed by crawler despite of the fact you closed those pages in robots.