Programming and Scripts
Coding, programming, scripts and integrations.

1
posted by saveta in Programming and Scripts 2 months ago

Blocking Bots/Spiders on Forum

Own several forums, one of which is hit with bots/crawlers all day everyday... What is the best way to block these guys? I got 121 crawlers right now hitting my site... I hate to think how much this slows my site down or how much bandwidth they are sucking down every month...

2 83
1

What are your thoughts? Sign Up or Log In

treefitty - 1 month ago
Using .htaccess to block bots or spiders might help...

#Example
BrowserMatchNoCase "SemrushBot" bots
Order Allow,Deny
Allow from ALL
Deny from env=bots

Adding the text above in the .htaccess file has worked for me in the past...

HTH
treefitty - 1 month ago
Also ran across this solution that you can add to block out nasty bots from Asia...

https://www.johnlarge.co.uk/blocking-aggressive-ch...

You can add the code below to your .htaccess file to block out Mb2345Browser, LieBaoFast, zh-CN, MicroMessenger, zh_CN, Kinza, typically found in the UserAgent... Should work until they change UserAgent...

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} Mb2345Browser|LieBaoFast|zh-CN|MicroMessenger|zh_CN|Kinza [NC]
RewriteRule ^ - [F,L]

HTH

community details

Programming and Scripts
0 subscribers
6 posts
9 months ago Started

Coding, programming, scripts and integrations.

Create A Post
Latest Posts