Memory issues with dynatrace 5.6 prod edition
Has anyone else seen an increase in CPU and more memory/GC churn when upgrading from dynatrace 5.5 to dynatrace 5.6? Our DT server is definitely slower and clocking more, and we are trying to pick away at things we can disable to help with the issue. Any suggestions (besides the obvious one of reducing the purepath size) would be much appreciated. We have reduced the purepaths (which still arent small, but no different than previously), enabled JDBC stmt aggregation, disabled UEM, reduced purepath max size, and turned off several system profiles that were suspect, etc.
Ideas?
-Todd Anderson
Hi Todd,
I have seen after the upgrade at a customer an increase of skipped non-analyzed PurePaths. The reason there is that they are using our mobile native agent and user actions from this agent that get dropped were not counted in previous releases.
Klaus
If you already have a ticket open then please make sure you also give them all the information that you posted here.
The support team knows these JVM Settings better than I do - so - they can tell you if there is anything "suspicious" about it.
I am worried about the skipped events. Are they skipped on the server, the collector, the agent? Have you checked your collectors as well? Are they under more load?
Have all components been upgraded to the release? or are some components (e.g: individual agents) still running on the prev. version?
Andy, is are there any recommended changes to dtserver.ini, aside from what is installed with 5.6? We also merged some customizations in, that I want to make sure cause no harm:
-XX:+UseCompressedOops
-XX:+OptimizeStringConcat
-XX:+UseStringCache
-XX:-UseBiasedLocking
-Dcom.dynatrace.diagnostics.memory.corridorUpperLimit=90
-Dcom.dynatrace.diagnostics.memory.corridorReductionArea=85
-Dcom.dynatrace.diagnostics.memory.corridorNeutralArea=75
-Dcom.dynatrace.repository.truncation.variant=DELETE FROM {table}
Hi Andy, thanks for the response. We did indeed increase the dtserver max heap size to the limit of 14GB (previously it was 12 GB). We dont see any longer purepaths than before, but we are seeing more skipped purepaths and skipped events. I've reduced the instrumentation so that the purepaths are actually trending a little smaller on average than before.
We do have a support ticket open, but I thought I'd ping the community to see if someone else had already experienced this and solved it.
I've noticed the change in the memory mgmt. In dynatrace 5.5, the tenured heap would be consumed and would not fluctuate much (not much GC churn). Now, in dynatrace 5.6, we have the tenured heap going up and down more, and much more GC suspension time.
Hi Todd
The recommendation is to actually give the server a little more memory because we switched to Java 7 which has a slightly different memory management. Here is what we say about this in the dynaTrace 5.6 Release Notes:
dynaTrace is now on Java 7
Due the fact that Oracle won't deliver additional security updates for Java 6, all dynaTrace components are now optimized for Java 7. Since Java 7 utilizes different memory management, it is recommended that you increase the maximum available memory of the dynaTrace Server and Collector by about 20%. The maximum recommended memory settings for the dynaTrace Server have been increased to 14GB.
If this doesnt help it would be interesting if you observe any other changes, e.g: do you see longer PurePaths, more Skipped Events, ... in your Server Health Dashboards as before? If that is the case I recommend opening a support ticket
