help needed - deep thread starvation and memory analysis on production
hello
we have an .net application (client/server) that have a serious memory leak problem causing them to stop responding and for the users to get
the 4xx http messages.
the application is based on .net WCF services (on the servers) .
the servers are virtual based on VMWare - configured as shared use of the infrastracture capabilities.
each of them is designed to handle up to 16GB of RAM (to all the instances of the worker processes) and using a virtual 8 core CPU.
each of the servers is hosting a single application pool but with 5-10 instances of worker processes.
each of the worker processes is handling user requests by opening a paralel calls to an oracle database.
the worker processes internal threads are controled through .net 4.0 thread monitors and contacting the database through dbpool.
we use the dal of the enterprise library ... so we didnt overriden any of the basic db connection handling (no close() or dispose()).
we see in DT client that there are a lot of thread "sync" time and the memory is not cleaned (GC) till its way out of the worker process 1gb limit
(we can see up tp 2.5GB of memmory taken by the app)
we are monitoring the app for several days and while it keeps on crashing (especially on full load)... we can see that one thread is occuping
the CPU for 13% ... the rest of the threads are on WAIT status and never get CPU time ....
while this is hapaning ... we can see an increament on memory and than "crash".
so now for our question....
do any of you have any ideas how we can identify such an illusive application error ?
maybe you all have any suggestion on directing us upon monitoring this solution.
we have used already the threads deshlet and we are dumping memory just when they reach above 90% of memory...
but still.... no pinpoint for the exact cause.
thank you for your help.... the application is a very critical and couse us not just PR problems but mostly money transfer and handling ones.
therefore ... i'll very mush appriciate your help and cooperation
Hi Eyal,
Some more thorough investigation is needed to analyse this problem.
This seems like something that the Dynatrace Enablement Services team can help you out with.
I have forwarded your query to your Customer Success Manager who will get in touch with you shortly.
KR,
Kristof
