• Forums
    • Public Forums
      • Community Connect
      • Dynatrace
        • Dynatrace Open Q&A
      • Application Monitoring & UEM
        • AppMon & UEM Open Q&A
      • Network Application Monitoring
        • NAM Open Q&A
  • Home /
  • Public Forums /
  • Application Monitoring & UEM /
  • AppMon & UEM Open Q&A /
avatar image
Question by Brendan B. · Apr 09, 2013 at 12:41 PM · web performance monitoring

Filtering out BOT traffic so doesn't get captured in UEM

Hi-

I have a customer (vegas.com) who needs to filter out BOT traffic so it doesn't capture meaningless information that consumes all their paid for capture. I know we can filter out by IP ranges but what about filtering by user agent from tealeaf as something like this?

Drop28=reqfield HTTP_USER_AGENT contains crawl
Drop29=reqfield HTTP_USER_AGENT contains bot
Drop30=reqfield HTTP_USER_AGENT contains slurp
Drop31=reqfield HTTP_USER_AGENT contains spider
Drop32=reqfield HTTP_USER_AGENT contains jeeves
Drop33=reqfield HTTP_USER_AGENT contains Sleuth
Drop34=reqfield HTTP_USER_AGENT contains Nikto
Drop35=reqfield HTTP_USER_AGENT contains Nessus
Drop36=reqfield HTTP_USER_AGENT contains Heritrix
Drop37=reqfield HTTP_USER_AGENT contains IPCheck

can we filter those out so they don't get captured somehow?

thank you
~Brendan

Comment

People who like this

0 Show 0
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

3 Replies

  • Sort: 
  • Most voted
  • Newest
  • Oldest
avatar image

Answer by Jesper G. · Feb 14, 2014 at 03:46 AM

Also I think that dealing with bots can go a little bit deeper.

For example in the Errors view the bot traffic tend to generate alot of 404s since they are trying to index pages that are not valid any more leading to a "background noice" that can drown-out real problems that might occur and which doesn't generate enough of a signal to rise above bot-traffic.

One way to handle it would be to configure the error categories similar to "HTTP 4XX Response (internal)" with an "exclude bot"  attribute or have user agent mapping rules able to configured, such that UA strings with "GoogleBot" could be excluded.

It's a bit separate so I'll make a separate RFE on that.

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Reinhard W. · Apr 10, 2013 at 04:39 AM

Brendan,

I assume bots do not eat up visit volume as they usually do not interpret JS...

That said I could imagine someone could hack something together to use the already existing "exclude" browser/versions in the UEM config.
I'm thinking of replacing the user Agent string on the webserver side (Apache mod_header conditional config) to set it to something very exotic (like Opera version 0.1) and then configure the UEM restriction to exclude that browser.

I also tried to set the do-not-track header on Apache that way, which basically works if the signal handling is completely done on the Java/.Net backend and not on the webserver (due to the order of apache modules beeing processed).

Reinhard

Comment

People who like this

0 Show 3 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Klaus E. ♦ · May 07, 2013 at 04:04 AM 0
Share

Hi Reinhard,

Some bots like Google eat up license volume as they execute JavaScript!

Klaus

avatar image Roman S. · May 07, 2013 at 04:10 AM 0
Share

Another use case would be in test. Customer has a test environment with a small UEM volume in order to test their app with UEM, ensuring usernames are captured and so on. Selenium tests running on that environment should not use up the UEM volume...

Right now the only workaround is to filter out IPs that Selenium is using and hoping they don't suddenly change.

Best, Roman

avatar image Brian W. Roman S. · Jun 07, 2013 at 02:12 PM 0
Share

Roman,
Another option is to override the useragent of the selenium browsers to identify themselves as a browser version that does not exist, for instance "Firefox/1500"
In UEM configuration, you can exclude firefox versions >= 1500 and exclude the selenium browser.

avatar image

Answer by Klaus E. · Apr 10, 2013 at 03:34 AM

Hi Brendan,

This would be an extension to the capturing restrictions that you currently can define for Browsers. There is currently no custom selection available just the predefined once.

Klaus

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

How to get started

First steps in the forum
Read Community User Guide
Best practices of using forum

NAM 2019 SP5 is available


Check the RHEL support added in the latest NAM service pack.

Learn more

LIVE WEBINAR

"Performance Clinic - Monitoring as a Self Service with Dynatrace"


JANUARY 15, 3:00 PM GMT / 10:00 AM ET

Register here

Follow this Question

Answers Answers and Comments

5 People are following this question.

avatar image avatar image avatar image avatar image avatar image

Related Questions

UEM visit concerning an Application load balanced on 2 DataCenters

User Action Details "URL" un-measurable?

UEM not working because of re-directs answer re-write rule syntax

Release of JavaScript supported by dtagent_xxxx js

Should I see these 409 errors in dT?

Forum Tags

dotnet mobile monitoring load iis 6.5 kubernetes mainframe rest api dashboard framework 7.0 appmon 7 health monitoring adk log monitoring services auto-detection uem webserver test automation license web performance monitoring ios nam probe collector migration mq web services knowledge sharing reports window java hybris javascript appmon sensors good to know extensions search 6.3+ server documentation easytravel web dashboard kibana system profile purelytics docker splunk 6.1 process groups account 7.2 rest dynatrace saas spa guardian appmon administration production user actions postgresql upgrade oneagent measures security Dynatrace Managed transactionflow technologies diagnostics user session monitoring unique users continuous delivery sharing configuration alerting NGINX splitting business transaction client 6.3 installation database scheduler apache mobileapp RUM php dashlet azure purepath agent 7.1 appmonsaas messagebroker nodejs 6.2 android sensor performance warehouse
  • Forums
  • Public Forums
    • Community Connect
    • Dynatrace
      • Dynatrace Open Q&A
    • Application Monitoring & UEM
      • AppMon & UEM Open Q&A
    • Network Application Monitoring
      • NAM Open Q&A