Could someone please tell me if the following rules in my robots.txt would allow google or NOT to crawl the following example links?
Allow: /search/
Disallow: /*?
it´s an e-commerce and I would like to understand if when searching in the product search bar, a dynamic link like these would be crawled or not:
https://www.whateverwebsite.de/search/?q=hello
https://www.whateverwebsite.de/category.html?form_new=18658
https://www.whateverwebsite.de/search/?q=grand&productFilter=motive_design%3AGeometric
Those links are generic examples, but I would really like to know if the Disallow:/*?
blocks this kind of links to be crawled on not since there is nothing between the "/" and the "?".
Thanks a lot in advance and I look forwards to some answers to keep learning 🙂
2
Answers
Your question is answered here: A Deeper Look At Robots.txt
But, what you also should consider is that conflicting rules are resolved by giving precedence to the longer rule. (compare here)
So, take care that the right rule gets precedence as you want it.
You should use robots.txt testing tool https://www.google.com/webmasters/tools/robots-testing-tool?utm_source=support.google.com/webmasters/&utm_medium=referral&utm_campaign=%206062598
to know what is being blocked or what is not being blocked by search engine.