Question 22439 in SEO Question posted in
Some basic information to get you started with this can be found here.

Disallow: /*? in robots.txt explanation needed – SEO

Could someone please tell me if the following rules in my robots.txt would allow google or NOT to crawl the following example links?

Allow: /search/

Disallow: /*?

it´s an e-commerce and I would like to understand if when searching in the product search bar, a dynamic link like these would be crawled or not:

https://www.whateverwebsite.de/search/?q=hello

https://www.whateverwebsite.de/category.html?form_new=18658

https://www.whateverwebsite.de/search/?q=grand&productFilter=motive_design%3AGeometric

Those links are generic examples, but I would really like to know if the Disallow:/*?
blocks this kind of links to be crawled on not since there is nothing between the "/" and the "?".

Thanks a lot in advance and I look forwards to some answers to keep learning 🙂

Tags:

2

Answers


  1. 0

    Your question is answered here: A Deeper Look At Robots.txt

    Disallow: /*? # block any URL that includes a ?

    But, what you also should consider is that conflicting rules are resolved by giving precedence to the longer rule. (compare here)

    So, take care that the right rule gets precedence as you want it.

    Login or Signup to reply.
  2. 0

    You should use robots.txt testing tool https://www.google.com/webmasters/tools/robots-testing-tool?utm_source=support.google.com/webmasters/&utm_medium=referral&utm_campaign=%206062598
    to know what is being blocked or what is not being blocked by search engine.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search