Dynatrace Server - Handling Load Tests Load
Hello guys,
We do some pretty heavy load testing here before big events such as Black Friday that is coming up in a few months, and we are running into issues where the Dynatrace Server is unable to handle all the requests during the tests During the tests we have more than 3 million purepaths per 5 minutes.
The issue is that the server uses all memory and starts executing a lot of Garbage Collection, pretty much losing all purepaths that arrive.
So my question is, how can I increase the server memory past the X-Large definition? We are guaranteed not to be able to handle Black Friday traffic, Those purepaths come from very few agents, and we cannot afford to filter out even more data!
We will upgrade the CPU to 24 cores and we can get pretty much as much memory as we want.
Thanks!
Hi. I would first look into reducing your PurePath length. Its a bit hard to say what your average size is as the chart shows the 10000 max size. But - please make sure you have done everything to reduce PP size. Also - make sure to exclude PurePaths that you dont really need, e.g: static URL requests, keep-alives, ... -> you can exclude certain URLs from the Web SErver and Servlet/ASP.NET Sensors. Also - make sure you have Database and Exception Aggregation enabled. That will also bring your PP size down.
If this doesnt help I would also check defined Measures and BTs. If you have a lot of Measures that use complex RegEx's it could impact your Real Time Analyzer. So - double check these as well.
If that all doesnt help I suggest you reach out to our support team. They can help you to go beyond XLarge
Andi
Thanks for the answers! We will try to optimize the environment as much as possible, 90% of the purepaths are from our "Price" API, that is pretty much 2 or 3 Mongo queries each, so the PP size for those guys is very low.
But still, the average PP length is very high, we will try to reduce it!
Another thing we might do is disable the BTs during Load Testing since we do not need them.
Aggregation is Enabled for ADO.NET and Exceptions, no static content being captured
Hi David,
3 million PP/5 min equates to 10k PP/s. This is way above the max advertised throughput of ~5k PP/s.
A lot will depend on the PP size as the advertised 5k PP/s assumes an average PP size of 100. The max transaction count actually refers to the maximum amount of instrumented nodes per second the server can handle. This value is roughly ~500k for 6.2. So you multiply the max PPS of 5k with average size of 100 and you get to that number. https://community.dynatrace.com/community/display/...
If you can get the average size down to let's say 50 (ideally even lower), the server might be able to handle the load (10k PP/s * 50 = 500k MPS). Increasing the number of cores, however, will be crucial as that defines how many Real-Time Analyzer threads there will be (# threads = # non-hyperthreaded cores/2) and thus will increase the throughput of the server.
Also make sure that the server is actually able to write the PP to the disk fast enough. If you are using a visualization layer this might severely inhibit the writing performance. Best is to do an IOPS test and compare it against the sizing guide.
Hope this helps.
KR, Kristof
Seems @Andreas Grabnerand I were in a race and I lost :-)
Still valuable information, thanks @Kristof Renders
