My boss is having me look at various ways to improve our site's SEO and I've been doing some research on it. I'm aware that search engines like mobile-friendly sites and I used Google's Webmaster Tools, finding that it considers…
Several domains are configured as add-ons to my primary hosting account (shared hosting). The directory structure looks like this (primary domain is example.com): public_html (example.com) _sub ex1 --> displayed as example-realtor.com ex2 --> displayed as example-author.com ex3 --> displayed as…
These days robots.txt became an important tool for SEO in websites. Through this file, web developers says crawler robots to check and not to check specific paths. But on the other hand, there are many secret and important directories and…
First, i've searched by robots.txt for Wordpress, but, no one told me where is this file. So, I read that the robots.txt in Wordpress is virtual. Ok, no problem. But, where i find this to edit? My Wordpress is allowing…
I have read an article about ignoring the robots from some url in my ASP MVC.NET project. In his article author said that we should add some action in some off controllers like this. In this example he adds the…
Having to manage a multilingual site, where users are redirected to a local version of site, like myBurger.com/en // for users from US, UK, etc.. myBurger.com/fr // for users from France, Swiss, etc... How should be organized the robots.txt file…
I want my site to be indexed in search engines except few sub-directories. Following are my robots.txt settings: robots.txt in the root directory User-agent: * Allow: / Separate robots.txt in the sub-directory (to be excluded) User-agent: * Disallow: / Is…
In Sitecore I'm trying to set up a way for our client to modify the robots.txt file from the content tree. I am attempting to set up a MVC controller action that is mappled to route "robots.txt" and will return…
We are getting bruteforce attacked on our sites and I am afraid to ban the IP's as they may be rotating IP's or legitimate users at some point in there life span. I would like to block all unknown bots…