I have tried using a code from a guy in github but it didn't work for me
## Add here all user agents that are to be blocked.
map $http_user_agent $bad_bot {
default 0;
~*^Lynx 0; # Let Lynx go through
libwww-perl 1;
~*(?i)(httrack|htmlparser|libwww|JikeSpider|proximic|Sosospider|Baiduspider|msnbot|BBBike|WWWOFFLE|Widow|SuperHTTP|BlackWidow|HTTrack|Java|Pixray|CPython|Spinn3r|Abonti|MSIECrawler|Baiduspider|Yandex|Siteimprove|Aboundex|80legs|360Spider|^Java|Cogentbot|^Alexibot|^asterias|^attach|^BackDoorBot|^BackWeb|Bandit|^BatchFTP|^Bigfoot|^Black.Hole|^BlackWidow|^BlowFish|^BotALot|Buddy|^BuiltBotTough|^Bullseye|^BunnySlippers|^Cegbfeieh|^CheeseBot|^CherryPicker|^ChinaClaw|Collector|Copier|^CopyRightCheck|^cosmos|^Crescent|^Custo|^AIBOT) 1;
}


## Add here all referrers that are to blocked.
map $http_referer $bad_referer {
default 0;
~(?i)(adult|babes|click|diamond|forsale|girl|jewelry|love|nudit|organic|poker|porn|poweroversoftware|sex|teen|webcam|zippo|casino|replica) 1;
}


## Add here all hosts that should be spared any referrer checking.
geo $bad_referer {
127.0.0.1 0;
192.168.1.0/24 0;
}

Do I have to install any addon ?
Anyone have any script to block bad bot ?

Hi there, there are a LOT of very bad nginx bad bot blocker scripts out there. Most of them very badly coded and simply copy and pasted from other people's scripts, which were copy and pasted from other people's scripts.

When I started building my own bad bot and bad referer blocker I started with the very same script you came across above, then I came across better one's but to be honest they were all rather messy and poorly structured and coded.

I wrote my own blocker from the ground up over a period of 4 months. I started with a clean slate and built it purely out of my own server logs from about 28 live web sites which all get a lot of daily traffic. I tested it thoroughly for nearly 3 months before releasing it publicly and it is in live operation on both of my Nginx servers and works like an absolute charm. Not only is is blocking out all the bad bots and referers but over a period of 4 months in live operation on my own servers it has actually increased traffic on all my sites across the board and has also increased ad revenue of the web sites where I am running Google Adsense, so it works and proves there was a lot of bad traffic coming into my sites which were stealing traffic.

I am also a photographer so I have a lot of image theft and image hot-linking to deal with so my blocker contains extensive amounts of web sites that hot link and steal images. I also built a DDOS function and rate limiting functions into it so it is quite a flexible script and places absolutely no overhead onto Nginx because it is centralised.

I also set up cron jobs which get emailed to me daily giving me all the latest top 500 referers from every web sites log file for the past 25,000 hits, so every morning I sit down for 10-15 minutes, go through those emails and update my blocker. So unless I have an off day it is almost updated daily as you will see from the commits on Github.

I also tested extensively for any false positives over the entire period I built it and I can assure you it is perfect. You could say I tested the living hell out of it which is exactly what I did because all the other scripts I first used had many false positives due to the copy and paste method in which they were built and I proved none of them had ever been thoroughly tested which is why I built my own.

You can grab it here and I look forward to your feedback. Please read all my notes, you can simply use my blocker as is and grab my regular updates and you should be more than covered or your can fork from it and create your own based on referers coming into your own logs.

Grab it from > https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker



Edited 2 time(s). Last edit at 10/19/2016 04:37AM by mitchellkrog.