Programming and Scripts
Coding, programming, scripts and integrations.

Own several forums, one of which is hit with bots/crawlers all day everyday... What is the best way to block these guys? I got 121 crawlers right now hitting my site... I hate to think how much this slows my site down or how much bandwidth they are sucking down every month...

1 2 793

What are your thoughts? Sign Up or Log In

treefitty - 1 year ago
Using .htaccess to block bots or spiders might help...

BrowserMatchNoCase "SemrushBot" bots
Order Allow,Deny
Allow from ALL
Deny from env=bots

Adding the text above in the .htaccess file has worked for me in the past...

treefitty - 1 year ago
Also ran across this solution that you can add to block out nasty bots from Asia...

You can add the code below to your .htaccess file to block out Mb2345Browser, LieBaoFast, zh-CN, MicroMessenger, zh_CN, Kinza, typically found in the UserAgent... Should work until they change UserAgent...

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} Mb2345Browser|LieBaoFast|zh-CN|MicroMessenger|zh_CN|Kinza [NC]
RewriteRule ^ - [F,L]


topic details

Programming and Scripts
0 subscribers
6 posts
2 years ago Started

Coding, programming, scripts and integrations.

Create A Post