Memory Leak Dump - No application specific class
Hi,
For a customer OOM event on one of the jvms (IBM Websphere) java 1.6 , created leak analysis thread dump.
While analyzing that memory dump cannot see any application related classes in different views. Hotspots tab showed close to 2 GB occupied by 'bytes' objects but cannot identify whether those are related to application or native server packages.
I have requested memory dump from customer to share, meanwhile want to know if any one came across this scenario, where application specific classes / methods not shown in memory dump ?
-Thanks
Ankur
Thanks for help Rick.
Let me carry out these steps, will get back to you.
-Thanks, Ankur
Hi Richard,
With 'Shortest path to root' I can see some classes related to application code (search with 'hdb') but still not able to relate which one is causing leak. (Application architecture is bit complex - 30-40 applications running in same jvm).
Memory dump show Byte classes creating Vector instances which occupying more and these vector instances point to ibm jars.
memory files are quite big, check if you are able to download from this link : ftp://ftp.compuware.com/pub/cso/singapore/outgoing/wasp FTP.zip
If not attached a ppt showing some screenshots. Memory Dump Screenshots.pptx
Please check if you can and let me know how to proceed.
Thanks, Ankur
Here's what I would do from here (can't download the session files but your screenshots contain everything I think I need to see).
- Put a memory sensor (through right-click -> Add Sensor Rules, or through the Sensors dialog in Edit System Profile) on com.ibm.mm.sdk.server.DKResultSetCursorICM class,
- configure the sensor properties to show the allocations on the PurePaths, then
- restart the app and take Selective Memory snapshots as the heap grows. This will tell you what the allocating method is for the parent objects.
- If that method is still not in their code, take a CPU sample (very high frequency for 30-60 seconds under load) and trace where that method is being called down the call stack to the customer's code.
Rick B
There are lots of fringe explanations for something like this:
- Undersized Heap
- Abusive Object allocation (so not a technical leak, just not enough memory to function)
- An actual honest-to-goodness bug in the container or third-party APIs
Most often, though, the "hotspots" view is the starting place to find an answer, rather than the answer itself. Do you mean that there is one byte[] listed in hotspots as having GC size 2 GB? That's a lot. Is there a similarly sized object in the Collections and Maps tab? If you right-click the byte[] and follow the references to root, do you see application code there? What sorts of classes are they? How are they being implemented by the application? Once you have more of these answers it should be more clear what is happening in the application.
HTH
Rick B
