Seeking details on Automation parameter definitions and metric calculations
We will possibly use Dynatrace in our unit testing framework as well.
to that end, the documentation is lacking on the definitions of the metrics and parameters in the Automation interface.
Some relevant questions are:
How is the "Corridor" calculated? Is that mean +/= Margin of error at 97% confidence interval?
what does the default value of 100% volatility mean? The impression is that it is a measure of variance, but how is volatility calculated?
Does the number of calibration runs apply only to the initial 5 measurements (runs)? or is it applied every time changes are accepted?
How can the system recognize a 'false positive'? It determines violations (negatives) by the actual result and the corridor definition. So then what does a false positive % mean and how is it calculated?
How do the "accept changes" and "accept changes from Here" functions affect the volatility and corridor calculations?
Thanks for your reply.
Thanks, Andreas. I would like to get some clarification on the Hint Text. Specifically, once the minimum number of calibration runs is reached, are changes automatically accepted for every subsequent improved run, or after every “Improved Runs” lower limit? For instance, if Improved Runs lower limit is set to 2, are changes accepted after every 2 consecutive improved runs or simply after each improved run? Thanks.
The setting for Auto-accept changes is irrespective of the settings for improved runs. So changes are accepted for every subsequent improved/degraded run.
As the hint text for Improved Runs says this setting influences the classification of test performance state only. But there is a copy-paste-error in the hint text. The text must be
The minimum number of consecutive violations of the lower limit for a test until it is considered improved
of course.
Hi John,
- The corridor is the (100-False Positive %)-% Confidence Interval of the Student's t-Distribution of a measure (see http://en.wikipedia.org/wiki/Student%27s_t-distribution) - since False Positive % is 1 by default, the default is the 99% confidence interval.
- Volatility is our name for the Coefficient of Variation: http://en.wikipedia.org/wiki/Coefficient_of_variation. By changing the Volatile % in the Test Automation Settings, you define how high this Coefficient of Variation has to be such that a Test is considered volatile.
- The calibration runs are applied every time changes are accepted.
- See 1 - the "False Positive %" is our term for setting the confidence interval.
- When accepting changes, the system behaves exactly as if the selected measurement was the first one ever observed; all existing values for standard deviation etc. are discarded and are only shown in the chart. When selecting "Accept Changes" instead of "Accept changes from here", the first measurement outside the corridor will be chosen for accepting changes.
Hope this helps,
Thomas
One more question on the Corridor Settings (see System Profile > Test Automation)… How does the “Auto-accept changes” work? That is, what is the trigger when this flag is checked? Is it the “Calibration Runs” or the ‘Degraded Runs” & “Improved Runs”?. Thanks.
This is the Hint Text you get in the System Profile for this checkbox: "Automatically accept changes in improved runs as soon as the minimum number of calibration runs has been reached."
Does this help?
Hello John,
sorry I can´t shoot out of the hip, but it´s Friday afternoon and I´ll need key people to answer your questions thoroughly.
May I ask for your patience to get back to you Monday (June 25th 2012 <g>)
Thanks
G.
