I'm trying to find out all those bots who are just doing key word harvesting and (for me) useless SEO selling - like MJ12bot or AhrefsBot. My site is something like 24 years old, most of time under Wordpress, but…
I have a bash script that sorts the highest hits on my server logs, and prints the IP address and user agent: cat /var/log/apache2/proxy.example.com.access.log | awk -F'|' '{print $5 $11}' | sort -n | uniq -c | sort -nr |…
Lately i'm facing some issues with a data mining bot, extracting data from my website everyday at certain hours of the day. This will not only waste my bandwidth but also giving wrong data to my google analytics. They usually…
I need to pass user agent value into front end. I can get this value using $_SERVER['HTTP_USER_AGENT'] and write it into front end. (Actually I will be using Mage::helper('core/http')->getHttpUserAgent(), but I think it's just a magento helper to call above…
I've been searching for npm packages but they all seem unmaintained and rely on the outdated user-agent databases. Is there a reliable and up-to-date package out there that helps me detect crawlers? (mostly from Google, Facebook,... for SEO) or if…
I remember when I was in college we went over some problem where there was a smart agent that was on a grid of squares and it had to clean the squares. It was awarded points for cleaning. It also…