Question posted in 
full disclaimer, I am not a programer, I am an SEO trying to learn how to not rely on my developer for every little question I have.
Currently my issue is this. I use Screaming Frog to crawl my sites to layout the page titles, meta descriptions, h1, h2, etc so I can more easily plan out my changes.
The other day I wanted to run a report for my client and my own company website and got the following back.
So I know robots.txt is a way to make pages on your site but not have google crawl them. What I don’t know is why an entire site would have this message as opposed to just some pages.
Can anyone give advice on how to fix this or links to how to’s? I get this issue a lot and would like to educate myself so I don’t have to wait for someone else. I get these as well when I try indexing websites on Google Search Console.
Many Thanks
2
Answers
The robots.txt for your website has not been written properly if the intention is to index its content.
Or Screaming Frog might have a but if indeed the robots.txt file is written properly.
Or some webmaster decided the content was not worth indexing on Google or that bots would eat too much bandwidth (as in not being selective to restrict access).
Checking the current robots.txt file on that website, I see this content:
Which means the any page of that website is allowed to be crawled by any crawler (here the explanation of that file’s syntax: https://moz.com/learn/seo/robotstxt)
So the current file should not cause that error OP mentions. Seeing that this question is from June 30/2017 and the robots.txt file was last modified on Jul 11/2017, it seems since this question was opened the OP may have already fixed whatever problem they had.