seo Question posted in
Some basic information to get you started with this can be found here.

WordPress – Robotx.txt allows admin login? – SEO

First, i’ve searched by robots.txt for WordPress, but, no one told me where is this file. So, I read that the robots.txt in WordPress is virtual.

Ok, no problem. But, where i find this to edit? My WordPress is allowing the /author/admin and i don’t want this.

In dashboard, the only option for something like this is too general, like “block search engine to find this website” (something like this).

Anyone knows a solution?

Tags:

4

Answers


  1. 0

    You must create your own robots.txt file in root directory which will override the virtual file or use a plugin for that.

    Login or Signup to reply.
  2. 0

    There’s an undocumented robots_txt hook which you can use to customize the virtual robots.txt file. It takes two parameters: the full content of the robots.txt file that WordPress generated, and a boolean value showing whether the site is “public” (open to search engines) or not.

    You can add your own filter to this hook and add that URL to the virtual robots.txt with code like this in a plugin or your theme’s functions.php file:

    add_filter( 'robots_txt', 'my_robots_txt_filter' );
function my_robots_txt_filter( $robots_txt, $public ) {
    $robots_txt .= "nDisallow: /author/admin";
    return $robots_txt;
}
    Login or Signup to reply.
  3. 0

    If you are not familiar with robots.txt and you’re ok to install a good plugin you can try WP SEO by Yoast that include a tool for generate a custom robots.txt and much more. Here is the link: https://wordpress.org/plugins/wordpress-seo/
    Hope it helps.

    Login or Signup to reply.
  4. 0

    You should create your own robots.txt file and upload it to website root directory.

    Follow the following steps to create and upload into root folder:

    1. Open notepad

    2. Add the following text into this file, remember add also your website sitemap path 

      sitemap: http://www.yoursite.com/sitemap.xml 
User-agent: * 
Disallow: /cgi-bin/ 
Disallow: /wp-admin/ 
Disallow: /wp-includes/ 
Disallow: /wp-content/ 
Disallow: /archives/ 
disallow: /*?* 
Disallow: *?replytocom 
Disallow: /wp-* 
Disallow: /author 
Disallow: /comments/feed/ 
User-agent: Mediapartners-Google* 
Allow: / 
User-agent: Googlebot-Image 
Allow: /wp-content/uploads/ 
Allow: / 
User-agent: Googlebot-Mobile 
Allow: /

    3. Save as robots.txt

    4. Open your website root directory

    5. Copy and paste this file into root directory

      In that way your robots.txt file will be updated and will be shown in directory.

    6. Finally go to webmaster account in robots.txt section. You can test that file and you can submit that file too.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search