I built out a nextjs 13 webapp using clerk for auth. The issue is that the homepage of the web app isn't discoverable on Google. when i Test Live URL, this is how it looks: enter image description here I've…
In a Vue (3.4) app, I have created a robots.txt file at the root of my folder. I have deployed my website with the robots.txt file and I am unable to find it when typing the URL https://www.example.com/robots.txt/ I get…
Let's say you have a website hosted on example.org. That website has a single page, whose content is static if the client requesting is not logged in, but dynamic (according to the logged client) if the client requesting it is…
Disallowing /wp-includes, /wp-content/plugins (cache and themes) affects my SEO score? I want to have a clear SEO analyze for my website, and I want to be sure that disallowing some WP folders doesn't affect my SEO. Should I keep the…
I want to add robots.txt to my Laravel project but robots.txt packages I found are not compatible with Laravel 9+ so if you know there is any tutorial or package for latest version of Laravel, please share. Thanks.
Could someone please tell me if the following rules in my robots.txt would allow google or NOT to crawl the following example links? Allow: /search/ Disallow: /*? it´s an e-commerce and I would like to understand if when searching in…
I'm using nginx and pm2 to serve a nuxt app (SSR). I tried to use the sitemap module and the robot.txt module. The sitemap module works fine and is accessible at /sitemap.xml. However the robots.txt file is not accessible through…
I have a website www.example.com. So when I access robots.txt file like www.example.com/robots.txt and it shows multiple lines text which is created/prepared by SEO team. So there is a sub domain assets.example.com which is pointing to CloudFront. When I access…
I have issues with AspiegelBot crawling one of the sites on a server, this results in a lot of cores getting used up. I've been trying to block the bot in both in the sites htaccess with no sucess. The…
I have pages that end in a search?=keyword and I want Google to prevent from crawling these pages. Example pages are below. I was wondering if I put a Disallow: /search in the robots.txt, will that work? Example URL's: http://www.website.com/search?=neymar…