• Forums
    • Public Forums
      • Community Connect
      • Dynatrace
        • Dynatrace Open Q&A
      • Application Monitoring & UEM
        • AppMon & UEM Open Q&A
      • Network Application Monitoring
        • NAM Open Q&A
  • Home /
  • Public Forums /
  • Application Monitoring & UEM /
  • AppMon & UEM Open Q&A /
avatar image
Question by Daniele D. · Oct 11, 2014 at 02:48 AM ·

Skipping data generated by a specific user agent

Hi,

I found lots of web requests (and purepaths) generated by a crawler. The requests addresses many different URIs. I learned the name of the crawler by looking at the user agent of the web request.

I would like Dynatrace to skip everything that is generated by that crawler at the source. That is, I would like Dynatrace to discard any purepaths that are being generated by that crawler. 

Is there a way to do it, please?

Thank you,

Daniele

Comment

People who like this

0 Show 0
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

3 Replies

  • Sort: 
  • Most voted
  • Newest
  • Oldest
avatar image

Answer by James D. · Oct 11, 2014 at 06:34 AM

Daniele,

While you can't ignore this in the sensor configuration, you could create a business transaction that ignores that specific user agent. Or, you could even modify your built in "Web Page Request" business transaction.

Safe Way -

I would create a Business Transaction Filter based on the crawler user-agent. I would then duplicate the Web Page Requests business transaction. From here I would apply this as the business transaction that drives my Monitoring Dashboard for my web application and remove the current Web Page Requests BT. This would remove them from this view and if you created other dashboards or views based on the Web Page Requests BT you could drop this in as a replacement.

Unsafe Way -

I would create the same  BT filter based on the crawler user-agent and apply that filter to your built in Web Page Requests. This would instantly stop all reporting of those specific PPs for any dashboards which use that BT. 

!!WARNING!! This will change a built in business transaction. While it will give the desired result it *could* mess up the existing BT and  *could* break other dashboards.

 

I had intended to post an image step by step but I am unable to for permission reasons.

 

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Daniele D. · Oct 11, 2014 at 05:54 AM

Hi Srikar,

thanks for replying. The point is, teh URL addressed by the crawler and the ones one can address are the same. It would be great only if I could tell the web server and the servlet sensor to exclude all the URIs coming from a specific user-agent. 

If this is not possible, no option.

Thank you,

Daniele

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Srikar M. · Oct 11, 2014 at 03:15 AM

Hi Daniele,

I dont think you can configure dynaTrace to not capture server side purepaths based on a 'user agent'...however you can define exclusions based on the uri (I understand you are seeing many variations..is there anything under the query string that appears common?)...I would create the exclude rules at the web and app tier by editing 'web server' and 'servlet' sensors on the respective tiers. If you want these requests not to be included in reporting then that is doable by creating a business transaction where you create a filter on user agent.

Perhaps this can be an rfe.

Thanks,

Srikar

 

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

How to get started

First steps in the forum
Read Community User Guide
Best practices of using forum

NAM 2019 SP5 is available


Check the RHEL support added in the latest NAM service pack.

Learn more

LIVE WEBINAR

"Performance Clinic - Monitoring as a Self Service with Dynatrace"


JANUARY 15, 3:00 PM GMT / 10:00 AM ET

Register here

Follow this Question

Answers Answers and Comments

1 Person is following this question.

avatar image

Forum Tags

dotnet mobile monitoring load iis 6.5 kubernetes mainframe rest api dashboard framework 7.0 appmon 7 health monitoring adk log monitoring services auto-detection uem webserver test automation license web performance monitoring ios nam probe collector migration mq web services knowledge sharing reports window java hybris javascript appmon sensors good to know extensions search 6.3+ server documentation easytravel web dashboard kibana system profile purelytics docker splunk 6.1 process groups account 7.2 rest dynatrace saas spa guardian appmon administration production user actions postgresql upgrade oneagent measures security Dynatrace Managed transactionflow technologies diagnostics user session monitoring unique users continuous delivery sharing configuration alerting NGINX splitting business transaction client 6.3 installation database scheduler apache mobileapp RUM php dashlet azure purepath agent 7.1 appmonsaas messagebroker nodejs 6.2 android sensor performance warehouse
  • Forums
  • Public Forums
    • Community Connect
    • Dynatrace
      • Dynatrace Open Q&A
    • Application Monitoring & UEM
      • AppMon & UEM Open Q&A
    • Network Application Monitoring
      • NAM Open Q&A