time spent on native java method park
Hi everybody.
Months ago my application's response time starts to grow up sometimes. Now there is a lot of transactions with high response time. For that reason I tried Dynatrace and I figured out that all spent time was in sun.misc.Unsafe.park() which is a native java method.
Attach you have both screenshots and sessions showing that problem. These are 2 random transactions where you can see time spent in park method (the one which spent 60secs is due to connectionTimeout=60000 in Tomcat's AJP connector so yes, we have also an Apache).
I need a hint or something to fix that problem.
Thanks in advance!
Hi Christian.
We tried to change maxThreads on Tomcat's AJP connector with no results. We also try to use a Executor with the same result. In that post I understand that they suggest to not use maxThreads but let Tomcat to handle the pool, right?
It's weird that Tomcat doesn't use many threads despite the fact that there is a lot of requests. Besides, I don't understand why Tomcat's thread are waiting in the middle of a transaction or how can I determine where is that thread or what was it doing before getting parked :(
Hi Josepth.
1) Attach you have "show all nodes" clicked but It ends in the park method too. Looking at the tree, and I don't understand if there is anything
2) We have a lot of traffic, that's correct. Also, our System's team says that our in/out usage of Internet is correct.
Is there any other thing that we miss to figure out where is this thread parked?
you could e.g. look at this mailing list entry: https://mail-archives.apache.org/mod_mbox/tomcat-u...
they seemed to have the exact same problem as you and it was also related to the network. however they also made a suggestion around configuring the tomcat threadpools.
HTH, Christian
Javi,
Another way to look at this type of problem is (in the purepath dashlet), select the "show All Nodes" link that is located on the right side below the Purehapth Hotspot area. This will expand the call tree to include all nodes, not just the relavent nodes. With this additional perspective, you can see that the transaction is getting a request parameter, and getting hung up when trying to read in those bytes. I would take a look at the health of the Tomcat instance at this time and see if perhaps it is running out of some type of resource.
As an alternate approach, you can also right click on this purepath and select Drilldown -> Method Hotspots. This is another way to view a more complete call tree.
After inspecting the Infrastructure metrics, I also notice that there seems to be a lot of traffic going in/out of the NIC card(s). 28MB/sec. I would check to see if that is what you expect during this activity period.
Also, upon inspecting the Tomcat3 JVM, verify the Transaction rate is what you expect, ~500/sec.
also copying my post over from the other thread:
hi Javi,
Unsafe.park() is to my knowledge (and also in your case) mostly used by LockSupport.park(). and in those cases it's basically always some usage of the java.concurrent synchronization/locking structures. so this basically just means: some resource is waiting for another resource to get available.
in your case there's a simple way to get some more information: click on the "Show all nodes" on the right above the PurePath tree. I don't know anything about this application, but what I can see/assume from there:
- there is at least one Apache Coyote AJP call (org.apache.coyote.ajp.AbstractAjpProcessor.readMessage()) which seems to read for a very long time (29s/58s). further down the callstack you see a CountDownLatch indicating that it's waiting for something. I don't know the implementation, so I don't know if it's waiting for some resource to get free or just for the network/disk to return some data. you might want to instrument more in this direction to get more data about this.
the suggestions about multi-threading and thread dumps above are also valid and you should absolutely check if you can spot some locking issues in a thread dump.
HTH, Christian
Hi Christian.
Is there any way to know where or why is this thread parked? We have a lot of transactions and there is only a few of them that reach these high response times so our applications is never hung at all but these few transactions (it happens at every time, every day).
I would like to know what is CountDownLatch waiting for, as you said. Our thread dumps don't tell us anything about blocked threads (you can easily see that in the attached session of above post).
Hi Ari.
Attached you have a session with a ThreadDump done between 2016/03/10 16:17:43 to 2016/03/10 16:18:25.
Thanks a lot.
Hi Ari.
Thanks for reply. I did a thread dump but I don't know how to read into it. Besides, there is no peaks of concurency in our application. It seems that sometimes (many times during the day), some transaction has high response time (random) and it is always stuck in park().
Please, could you help me with that?
Can you post the thread dump here?
Its interesting that you don't see any peaks of concurrency in the app. My guess would have been that when there is a high amount of transactions going on, a lot of them get stuck (or 'parked') waiting on the same resource.
Hi Ari.
I cant see that post. It says "Access denied"
Thanks
Ah ok, I'll just copy what I wrote on there:-
I've actually seen a lot of this happen at my client site. Here is what we found out:-
unsafe.park is essentially the same as thread.wait except that it's using architecture specific code - the reason it's designated as 'unsafe'. unsafe is not made available publicly, but is used within java internal libraries where architecture specific code would offer significant optimization benefits - used a lot for thread pooling.
It’s very hard to be certain since I have very limited visibility into what you are seeing – but my guesses and suggestions would be as follows: A multi-threaded library is controlling access to some protected resource and the synchronization logic is never resolving. It like a loop with no exit condition. In this case the result is a bunch of “zombie” threads. Are your frontend response times tracking very closely with frontend concurrency? i.e. when concurrency goes up, response times get slower, and vice versa. If that is the case, the two most common cause for this are that the executable threads are waiting for access to: 1) A shared in-memory data collection (list, map, array) 2) A free thread in a DB connection pool (waiting to talk to the DB)
If possible, I would try taking a thread dump to see what's going on.
Hope this helps,
Ari
