We pushed out a 5.5.0.5562 patch this weekend in production however during the update the collector crashed (filesystem filled up). We got that worked out, restarted the collector and all our agents can connect to it now however in the agent logs some of them show they processed the update however in the client it still shows as the old version for all of them using that collector (5.5.0.5301). Is there a way to manually/force the update to take effect in the agents? Or could this still be a collector issue. I can reinstall the collector fairly easily but that has to wait until the weekend and still would be crossing my fingers the agents are updated automatically as well.
Answer by Michael C. ·
Sure, well here's the obvious ones:
2013-12-14 09:21:04 WARNING [SessionHandshakeRequest] session initializer error: No space left on device: com.dynatrace.diagnostics.communication.tcp.session.SessionHandshakeRequest executeR
equestResponse:87
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:282)
at com.dynatrace.diagnostics.sdk.io.DataInputStreamDataInput.readStream(DataInputStreamDataInput.java:198)
at com.dynatrace.diagnostics2013-12-14 09:21:04 WARNING [SessionHandshakeRequest] session initializer error: No space left on device: com.dynatrace.diagnostics.communication.tcp.sess
ion.SessionHandshakeRequest executeRequestResponse:87
java.io.IOException: No space left on device
But then subsequent restarts gave these here & there:
2013-12-14 10:12:54 SEVERE [ThrowableHandler] Uncaught exception in thread "Communication-Thread-0@main": illegal byte array size 16777216 (max. byte array size is 10485760): com.dynatrace.d
iagnostics.util.modern.ThrowableHandler uncaughtException:50
java.lang.IllegalArgumentException: illegal byte array size 16777216 (max. byte array size is 10485760)
at com.dynatrace.diagnostics.sdk.io.DataInputStreamDataInput.readByteArray(DataInputStreamDataInput.java:329)
at com.dynatrace.diagnostics.communication.tcp.connection.DefaultSocketHeaderReaderWriter30.readHeader(DefaultSocketHeaderReaderWriter30.java:51)
at com.dynatrace.diagnostics.communication.tcp.connection.DefaultSocketHeaderReaderWriter30.readHeader(DefaultSocketHeaderReaderWriter30.java:24)
at com.dynatrace.diagnostics.communication.tcp.socket.DefaultSocketHandlerRunnable.handleRequest(DefaultSocketHandlerRunnable.java:65)
at com.dynatrace.diagnostics.communication.tcp.socket.DefaultSocketHandlerRunnable.execute(DefaultSocketHandlerRunnable.java:55)
at com.dynatrace.diagnostics.util.threads.StatusRunnable.run(StatusRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Answer by Michael C. ·
Just an update: We ended up completely reinstalling the collector in question and appears to have done the trick. We will be restarting the rest of our applications tonight after hours to see if they all fall in line ( I restarted one small one as a test and it worked). There must have just been something not quite right after the filesystem filled up and it crashed.
Thanks for the update. Would be interesting to inspect the collector log file of the "crashed" collector. If you still have it somewhere you may want to take a look at it and see whether it contains any log messages that indicate a problem. This would be valuable feedback for our engineering team
Answer by Andreas G. ·
If the server and collector are both updated and run with the latest version - and the agent doesnt correctly update to the fixpack version even after a restart I suggest opening a support ticket. A support engineer can then have a closer look at your problem.
Answer by Michael C. ·
(the reason this is even a significant issue is that this patch has a bug fix in it we need on some of our Prod webservers)
Answer by Michael C. ·
It shows 5562 (the updated version). It was bounced last night (command line) and actually did it again this morning via the console.
Answer by Andreas G. ·
In your server settings dialog you see an overview of all connected collectors. can you check which version of the collector you are running right now? Maybe you need to bounce your collectors as well so that they get properly upgraded
Answer by Michael C. ·
Hi Andreas,
We did bounce our apps, and they did then connect back to the collector however still show 5301. However I can see in their agent log the following:
2013-12-15 20:40:08 [40eeb940] info [native] Server requests us to use Agent 5.5.0.5562 with a hash of a0007c29349ce008e9c9e3bce4279356
2013-12-15 20:40:08 [40eeb940] info [native] Agent already available locally
But then after that in the next log:
2013-12-15 20:40:10 [40eeb940] info [java ] Java Agent Version .......... 5.5.0.5301, Agent15
Answer by Andreas G. ·
The agents will automatically get updated when they start. So - rebouncing your application should do the trick.
JANUARY 15, 3:00 PM GMT / 10:00 AM ET