• Forums
    • Public Forums
      • Community Connect
      • Dynatrace
        • Dynatrace Open Q&A
      • Application Monitoring & UEM
        • AppMon & UEM Open Q&A
      • Network Application Monitoring
        • NAM Open Q&A
  • Home
  • Public Forums
  • Application Monitoring & UEM
  • AppMon & UEM Open Q&A
avatar image
Question by Javi M. · Mar 11, 2016 at 08:29 AM · agent java business transaction apache

time spent on native java method park

Hi everybody.

Months ago my application's response time starts to grow up sometimes. Now there is a lot of transactions with high response time. For that reason I tried Dynatrace and I figured out that all spent time was in sun.misc.Unsafe.park() which is a native java method.

Attach you have both screenshots and sessions showing that problem. These are 2 random transactions where you can see time spent in park method (the one which spent 60secs is due to connectionTimeout=60000 in Tomcat's AJP connector so yes, we have also an Apache).

I need a hint or something to fix that problem.

Thanks in advance!

dynatrace-similareslistadopisos.png (180.8 KiB)
dynatrace-timeouttomcat.png (147.5 KiB)
dynatrace-similareslistadopisos.dts (888.9 KiB)
dynatrace-timeouttomcat.dts (857.8 KiB)
Comment

People who like this

0 Show 0
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

9 Replies

  • Sort: 
  • Most voted
  • Newest
  • Oldest
avatar image

Answer by Javi M. · Mar 14, 2016 at 04:16 PM

Hi Christian.

We tried to change maxThreads on Tomcat's AJP connector with no results. We also try to use a Executor with the same result. In that post I understand that they suggest to not use maxThreads but let Tomcat to handle the pool, right?

It's weird that Tomcat doesn't use many threads despite the fact that there is a lot of requests. Besides, I don't understand why Tomcat's thread are waiting in the middle of a transaction or how can I determine where is that thread or what was it doing before getting parked :(

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Javi M. · Mar 14, 2016 at 11:03 AM

Hi Josepth.

1) Attach you have "show all nodes" clicked but It ends in the park method too. Looking at the tree, and I don't understand if there is anything

showallnodes.png

2) We have a lot of traffic, that's correct. Also, our System's team says that our in/out usage of Internet is correct.

Is there any other thing that we miss to figure out where is this thread parked?


showallnodes.png (214.5 KiB)
showallnodes.png (214.5 KiB)
Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Christian S. · Mar 14, 2016 at 03:58 PM 0
Share

you could e.g. look at this mailing list entry: https://mail-archives.apache.org/mod_mbox/tomcat-u...

they seemed to have the exact same problem as you and it was also related to the network. however they also made a suggestion around configuring the tomcat threadpools.

HTH, Christian

avatar image

Answer by Joe H. · Mar 11, 2016 at 03:57 PM

Javi,

Another way to look at this type of problem is (in the purepath dashlet), select the "show All Nodes" link that is located on the right side below the Purehapth Hotspot area. This will expand the call tree to include all nodes, not just the relavent nodes. With this additional perspective, you can see that the transaction is getting a request parameter, and getting hung up when trying to read in those bytes. I would take a look at the health of the Tomcat instance at this time and see if perhaps it is running out of some type of resource.

As an alternate approach, you can also right click on this purepath and select Drilldown -> Method Hotspots. This is another way to view a more complete call tree.

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Joe H. · Mar 11, 2016 at 04:08 PM 0
Share

After inspecting the Infrastructure metrics, I also notice that there seems to be a lot of traffic going in/out of the NIC card(s). 28MB/sec. I would check to see if that is what you expect during this activity period.

Also, upon inspecting the Tomcat3 JVM, verify the Transaction rate is what you expect, ~500/sec.

avatar image

Answer by Christian S. · Mar 11, 2016 at 02:27 PM

also copying my post over from the other thread:

hi Javi,

Unsafe.park() is to my knowledge (and also in your case) mostly used by LockSupport.park(). and in those cases it's basically always some usage of the java.concurrent synchronization/locking structures. so this basically just means: some resource is waiting for another resource to get available.

in your case there's a simple way to get some more information: click on the "Show all nodes" on the right above the PurePath tree. I don't know anything about this application, but what I can see/assume from there:

  • there is at least one Apache Coyote AJP call (org.apache.coyote.ajp.AbstractAjpProcessor.readMessage()) which seems to read for a very long time (29s/58s). further down the callstack you see a CountDownLatch indicating that it's waiting for something. I don't know the implementation, so I don't know if it's waiting for some resource to get free or just for the network/disk to return some data. you might want to instrument more in this direction to get more data about this.

the suggestions about multi-threading and thread dumps above are also valid and you should absolutely check if you can spot some locking issues in a thread dump.

HTH, Christian

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Javi M. · Mar 14, 2016 at 11:08 AM 0
Share

Hi Christian.

Is there any way to know where or why is this thread parked? We have a lot of transactions and there is only a few of them that reach these high response times so our applications is never hung at all but these few transactions (it happens at every time, every day).

I would like to know what is CountDownLatch waiting for, as you said. Our thread dumps don't tell us anything about blocked threads (you can easily see that in the attached session of above post).

avatar image

Answer by Javi M. · Mar 11, 2016 at 02:03 PM

Sorry but I forgot to attach session file

threaddump.dts


threaddump.dts (407.6 KiB)
Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Javi M. · Mar 11, 2016 at 02:02 PM

Hi Ari.

Attached you have a session with a ThreadDump done between 2016/03/10 16:17:43 to 2016/03/10 16:18:25.

Thanks a lot.

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Javi M. · Mar 11, 2016 at 01:42 PM

Hi Ari.

Thanks for reply. I did a thread dump but I don't know how to read into it. Besides, there is no peaks of concurency in our application. It seems that sometimes (many times during the day), some transaction has high response time (random) and it is always stuck in park().

Please, could you help me with that?

Comment

People who like this

0 Show 2 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Ari P. · Mar 11, 2016 at 01:57 PM 0
Share

Can you post the thread dump here?

Its interesting that you don't see any peaks of concurrency in the app. My guess would have been that when there is a high amount of transactions going on, a lot of them get stuck (or 'parked') waiting on the same resource.

avatar image Javi M. Ari P. · Mar 14, 2016 at 11:09 AM 0
Share

Yeah Ari, but how to figure out which resource is? I have attached a Dynatrace's session with thread dump

avatar image

Answer by Javi M. · Mar 11, 2016 at 01:20 PM

Hi Ari.

I cant see that post. It says "Access denied"

Thanks

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Ari P. · Mar 11, 2016 at 01:24 PM 0
Share

Ah ok, I'll just copy what I wrote on there:-

I've actually seen a lot of this happen at my client site. Here is what we found out:-

unsafe.park is essentially the same as thread.wait except that it's using architecture specific code - the reason it's designated as 'unsafe'. unsafe is not made available publicly, but is used within java internal libraries where architecture specific code would offer significant optimization benefits - used a lot for thread pooling.

It’s very hard to be certain since I have very limited visibility into what you are seeing – but my guesses and suggestions would be as follows: A multi-threaded library is controlling access to some protected resource and the synchronization logic is never resolving. It like a loop with no exit condition. In this case the result is a bunch of “zombie” threads. Are your frontend response times tracking very closely with frontend concurrency? i.e. when concurrency goes up, response times get slower, and vice versa. If that is the case, the two most common cause for this are that the executable threads are waiting for access to: 1) A shared in-memory data collection (list, map, array) 2) A free thread in a DB connection pool (waiting to talk to the DB)

If possible, I would try taking a thread dump to see what's going on.

Hope this helps,

Ari

avatar image

Answer by Ari P. · Mar 11, 2016 at 01:04 PM

Please refer to this post -> https://answers.dynatrace.com/questions/146405/sun...

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

How to get started

First steps in the forum
Read Community User Guide
Best practices of using forum

NAM 2019 SP5 is available


Check the RHEL support added in the latest NAM service pack.

Learn more

LIVE WEBINAR

"Performance Clinic - Monitoring as a Self Service with Dynatrace"


JANUARY 15, 3:00 PM GMT / 10:00 AM ET

Register here

Follow this Question

Answers Answers and Comments

12 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

How to exclude an argument value in a Filter measure

Monitoring Apache Spark?

check active client ip on apach

Interfering with Ajax calls

Does Dynatrace Java agent support Sybase Brand Mobiliser?

Forum Tags

dotnet mobile monitoring load iis 6.5 kubernetes mainframe rest api dashboard framework 7.0 appmon 7 health monitoring adk log monitoring services auto-detection uem webserver test automation license web performance monitoring ios nam probe collector migration mq web services knowledge sharing reports window java hybris javascript appmon sensors good to know extensions search 6.3+ server documentation easytravel web dashboard kibana system profile purelytics docker splunk 6.1 process groups account 7.2 rest dynatrace saas spa guardian appmon administration production user actions postgresql upgrade oneagent measures security Dynatrace Managed transactionflow technologies diagnostics user session monitoring unique users continuous delivery sharing configuration alerting NGINX splitting business transaction client 6.3 installation database scheduler apache mobileapp RUM php dashlet azure purepath agent 7.1 appmonsaas messagebroker nodejs 6.2 android sensor performance warehouse
  • Forums
  • Public Forums
    • Community Connect
    • Dynatrace
      • Dynatrace Open Q&A
    • Application Monitoring & UEM
      • AppMon & UEM Open Q&A
    • Network Application Monitoring
      • NAM Open Q&A