skip to Main Content

Do search engine robots crawl my dynamically generated URLs? With this I mean html pages generated by php based upon GET variables in the url. The links would look like this:

http://www.mywebsite.com/view.php?name=something
http://www.mywebsite.com/view.php?name=somethingelse
http://www.mywebsite.com/view.php?name=something

I have tried crawling my website with a test crawler found here: http://robhammond.co/tools/seo-crawler but it only visits my view page once, with just one variable in the header.

Most of the content on my website is generated by these GET variables from the database so I would really like the search engines to crawl those pages.

2

Answers


  1. Some search engines do, and some don’t. Google for one does include dynamically generated pages: https://support.google.com/webmasters/answer/35769?hl=en

    Be sure to check your robots.txt file to ensure files you do not want the crawlers to see are blocked, and that files you do want indexed are not blocked.

    Also, ensure that all pages you want indexed are linked via other pages, that you have a sitemap, or submit individual URLs to the search engine(s) you want to index your site.

    Login or Signup to reply.
  2. Yes, search engines will crawl those pages, assuming they can find them. Best thing to do is to simply create links to those pages on your website, particularly accessible, or at least traversable from the home page.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search