Send alert if the count of specific metric is above threshold

[Malaik]

Dear Community,

I want to open discussion,

What is the best way to set a metric event for a pecific service/request:

Example: sent email if the count of the service A is above 500 in the last 1 Hour.

[radek_jasinski]

In general, it depends very much on the specific environment and the requirements for the metric.

To give two examples:

At one of my clients, when Azure monitoring metrics are exceeded (without OA), SMS messages are sent via Webhook and SMS gateway to the relevant administrators.

In another, when OA usage metrics are exceeded, a ticket is created in Kira and a simple script is run via webhook to clear the RAM.

*Care must also be taken to ensure that the sliding window is not too short, as any slowdown in data transfer from the OA to the DT server can cause false problems.

[RazTN7]

Hello @Malaik 

Normally I might go with something like this.

set threshold as 500 violate 1 time during the sliding window of 60min (not sure 60 dealerting will help here) 

Thanks

RN

[Malaik]

Thanks,

But Im not sure that the suggested solution will work.

What i want is not that this threshold (500) was met 1 time in 60mn but I want the count of that metric in 60mn will not exceed 500.

may but 1 time 200 and second time 400, in that case i should raise alert which will not work with that solution

[ChadTurner]

All great replies, and just to echo some of the main points, it really depends on the number of samples in that time frame. So you said 1 hour (60 Mins) so the next question is how many times should this occur in 60 mins that would then trigger you to take action?  too low gives you too many alerts, too high gives you to little alerts.

[Malaik]

Thanks Chad,

But Im not sure that the suggested solution will work.

What i want is not that this threshold (500) was met 1 time in 60mn but I want the count of that metric in 60mn will not exceed 500.

may but 1 time 200 and second time 400, in that case i should raise alert which will not work with that solution