Fine Tuning Incidents
Hi,
I have created BTM’s for few critical transactions in Dynatrace and I have created dashboard where I have shown incidents dashlets where if response time or error rate of these transactions breached its get highlighted on incident dashboard.
But my problem is that I have set incidents on Avg response time and percentage of failure rate which giving me false alert as if any 1 of the requests response time is high that automatically show me high response time and if in last 1 minute if I have only 1 request and that request gets error it shows 100% error. Please suggest how I can avoid such false alerts.
I know that I can put these transactions in base lining feature of Dynatrace but it will not give me response time and failure rate alerts for all BTMs separately which I want to highlight in dashboard.
Also when I read documentation I found sensitivity field for incidents which were available prior to DT 4 is there any way where I can enable that field to check feasibility of my requirement.
Please suggest.
Regards,
Amol Khawre
HI,
To handle this scenario I created BTM and I have added web request count in BTM for web Request count calculation but when I add this measure in BTM I'm not getting threshold breached incidents even after threshold violations.
Please find attachment for details.
Please suggest.
Amol.
Your Incident uses two different Conditions connected with OR. Is it possible that the other Measure exceeded the threshold and therefore triggered the Incident. So - please have a look at all the measures that you use in your incident definition
As for Percentiles: No - this is currently not supported
Hi,
Please suggest if we can set alerts on percentile of response time as average giving false alerts.
Amol.
Hi Andi,
Thanks for explanation of logic. Please find attachment for details which you asked and help me understand this issue.
Amol khawre
HI Andi,
Thanks for revert but I have already shared incident configuration details in previous attachement where I highlighted that I have used Avg aggregation in incidents but still Its not working for me.
Please suggest.
Amol
Can you create another chart but this time use a 10s aggregation. I am interested in the individual data points. Reason why this is interesting is becuase Incidents are evaluated by constantly looking at the full timewindow of your evaluation timeframe. Its like a "rolling window". So - it could be possible that a very high failure rate will push the Avg. number across your threshold if you analyze the right 5 minute time window.
Andi
Please suggest.
Amol Khawre
Hi Amol
Is it possible that you chart e.g: "Average %" - but in the incident you choose something like "Max"
Remember that in both Charting as well as in Incidents you have the option to specify which "Aggregation" dynaTrace shoudl use to evaluate a measure. We can do min, max, avg, count,
So - please double check the "Aggregation" setting in your incidents to align it what you see in the Chart.
Hi Richad,
I have chnaged config as you suggested but still its not working for me. Please find attached excel for details and help me to understand if I is their any gap in my understanding about Incidents.
Regards,
Amol Khawre
Please help.
Amol
Hi Amol,
Fine-tuning incidents comes down to balancing the condition measure aggregation with the evaluation timeframe.
For instance, if you want to soften your incident to reduce false positives, you may want to extend your existing "average" condition over a timeframe of 5 minutes so that there is certainly a problem before throwing the alert. Likewise if you have a lot of traffic and some of it is usually slow but you want to track slowness across the board, you may want to consider a "minimum" aggregation over your timeframe of 1 minute to be sure that all requests were slow.
Rick B
