i would like to know how the performance metrics are calculated.When run a 2 diffrent versions of same test i see some metrics generated like:
Volatility,Improved runs,Expected maximum (for purepath),degraded and improved runs etc...
Please can you help me on this.
Answer by Shafiz H. ·
Thanks Andi for the response.
In my case running some Units for Different build's for single run.even i could see them in Test results dashboard.Form pure path comparison i could see that variation in response time for some Unit test in the builds's when compared.
But in Test results dashboard i don't see any information showing that test is Improved or Degraded.
And even i see Expected Minimum and Expected Maximum time for selected test run in Pure paths.
To show the Test as Improved or Degraded do i need to change any settings in Tests automation under system profile.
Even please let me know how the Expected Minimum and Expected Maximum time for each Unit test pure path for example:
I have a test in
2nd run:Pure path value is 57354.31 MS
from the above i can see that 2nd Run is taking more time to execute.
But on what basis the Expected Minimum and Expected Maximum time are calculated.
Please let me know how to view the degraded and Improved runs inTest results dashlet that are executed only once for different build.
PFA the screenshots.
Answer by Andreas G. ·
I suggest you check out the documentation on Continuous Delivery and Test Automation. There are several doc pages that explain that these metrics come from the PurePaths and are associated with your tests. We also have a baselining of each Metric for each Test which allows us to calculate a "Performance Corridor". The first corridor will show up after we have at least 5 test results. After that we look whether the measures of the current tests falls within that corridor or outside. if outside it means that you have a violation. If you have 2 consecutive violations we consider the test as degraded. If a measure constantly produces different results we mark it volatile. But - check out the doc for more information on this
I also have a youtube tutorial where I discuss this feature: https://www.youtube.com/watch?v=Pla4dWeqJgI&list=PLqt2rd0eew1bmDn54E2_M2uvbhm_WxY_6&index=25
Test Automation with Jenkins Plugin 3 Answers