Skipping data generated by a specific user agent
Hi,
I found lots of web requests (and purepaths) generated by a crawler. The requests addresses many different URIs. I learned the name of the crawler by looking at the user agent of the web request.
I would like Dynatrace to skip everything that is generated by that crawler at the source. That is, I would like Dynatrace to discard any purepaths that are being generated by that crawler.
Is there a way to do it, please?
Thank you,
Daniele
Daniele,
While you can't ignore this in the sensor configuration, you could create a business transaction that ignores that specific user agent. Or, you could even modify your built in "Web Page Request" business transaction.
Safe Way -
I would create a Business Transaction Filter based on the crawler user-agent. I would then duplicate the Web Page Requests business transaction. From here I would apply this as the business transaction that drives my Monitoring Dashboard for my web application and remove the current Web Page Requests BT. This would remove them from this view and if you created other dashboards or views based on the Web Page Requests BT you could drop this in as a replacement.
Unsafe Way -
I would create the same BT filter based on the crawler user-agent and apply that filter to your built in Web Page Requests. This would instantly stop all reporting of those specific PPs for any dashboards which use that BT.
!!WARNING!! This will change a built in business transaction. While it will give the desired result it *could* mess up the existing BT and *could* break other dashboards.
I had intended to post an image step by step but I am unable to for permission reasons.
Hi Srikar,
thanks for replying. The point is, teh URL addressed by the crawler and the ones one can address are the same. It would be great only if I could tell the web server and the servlet sensor to exclude all the URIs coming from a specific user-agent.
If this is not possible, no option.
Thank you,
Daniele
Hi Daniele,
I dont think you can configure dynaTrace to not capture server side purepaths based on a 'user agent'...however you can define exclusions based on the uri (I understand you are seeing many variations..is there anything under the query string that appears common?)...I would create the exclude rules at the web and app tier by editing 'web server' and 'servlet' sensors on the respective tiers. If you want these requests not to be included in reporting then that is doable by creating a business transaction where you create a filter on user agent.
Perhaps this can be an rfe.
Thanks,
Srikar
