question

Pawandeep S. avatar image
Pawandeep S. asked ·

IO Error: Got minus one from a read call

Hi Team

We are receiving lots of incidents in DT related to Performance warehouse.

It seems PW connection is getting lost, thought to check with community here. What all we can do to mitigate the situation.

Pointing in this direction will be helpful.


Here is some samples.

Sample 1:

2019-10-05 02:14:55 UTC WARNING [CITestExecutionQueue] No connection to Performance Warehouse, test executions queue will not be processed until reconnection. [log message will be suppressed for 10 minutes]
2019-10-05 02:14:55 UTC INFO [CollectorPeer] Discarded db monitor event due to missing metrics, DB Agent: URDS_CRR_Oracle [log message will be suppressed for 1 minutes]
2019-10-05 02:15:03 UTC WARNING [AgentEventChunk] RECEIVED EVENT: 29 for subAgent: fb956f81 (masterAgent: 23aeaeee) that is not registered on server. Skipping events. Suppressing this warning for another 60000 ms.
2019-10-05 02:15:10 UTC INFO [JmxMetricsManager] DssTransactionCleanupTimer wakes up at 1570241710217. Size 0. Waited for lock: 0ms
2019-10-05 02:15:10 UTC INFO [JmxMetricsManager] Metrics calculation is done in 0ms. decodeCountPastFiveMinutes: 0, decodeTimeMsPastFiveMinutes: 0
2019-10-05 02:15:10 UTC INFO [AbstractRepository] [REPOSITORY-RECONNECT-THREAD] started trying to reconnect the performance warehouse ...
2019-10-05 02:15:16 UTC INFO [DelegatingDataSource] An exception of class java.sql.SQLRecoverableException was thrown when trying to get a connection from the data source. Message : IO Error: Got minus one from a read call, connect lapse 6983 ms., Authentication lapse 0 ms., code: 17002, retry counter: 3
2019-10-05 02:15:16 UTC WARNING [DeferredAnalysisCacheSink] Failed to put or update entry in AnalysisCache @ AnalysisCacheKey [systemProfileId=, analyzerType=slowDatabaseQueries, splittings={databaseAgent=UAT_DFC_IQ_ALSCOMARCHIVE_MSSQL}] [log message will be suppressed for 1 hours]: com.dynatrace.diagnostics.common.logging.ThrottlingLogger logViaJdk:156
com.dynatrace.diagnostics.repository.handlers.analysiscache.api.AnalysisCacheHandlerException: com.dynatrace.diagnostics.server.shared.exceptions.repository.RepositoryNotConnectedException: IO Error: Got minus one from a read call, connect lapse 6983 ms., Authentication lapse 0 ms., most likely because repository is not connected (UNABLE_TO_CONNECT).
at com.dynatrace.diagnostics.repository.handlers.analysiscache.AnalysisCacheConnectionExecutionCallbackAdapter.throwOccurredException(SourceFile:33)
at com.dynatrace.diagnostics.repository.handlers.analysiscache.DefaultAnalysisCacheHandler.get(SourceFile:560)
at com.dynatrace.diagnostics.repository.handlers.analysiscache.OracleAnalysisCacheHandler.get(SourceFile:19)
at com.dynatrace.diagnostics.server.analysiscache.store.DatabaseCacheStoreManager.get(SourceFile:173)
at com.dynatrace.diagnostics.server.analysiscache.store.DatabaseCacheStore.addOrUpdate(SourceFile:50)
at com.dynatrace.diagnostics.server.analysiscache.store.LazyDatabaseCacheStore.addOrUpdate(SourceFile:58)
at com.dynatrace.diagnostics.server.analysiscache.AnalysisCacheEntry.addOrMerge(SourceFile:64)
at com.dynatrace.diagnostics.server.analysiscache.AnalysisCache.putOrUpdate(SourceFile:159)
at com.dynatrace.diagnostics.server.analysiscache.DeferredAnalysisCacheSink$1.run(SourceFile:44)
at com.dynatrace.diagnostics.server.ScheduledThreadPoolExecutorWithExceptionListener$3.run(SourceFile:98)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.dynatrace.diagnostics.server.shared.exceptions.repository.RepositoryNotConnectedException: IO Error: Got minus one from a read call, connect lapse 6983 ms., Authentication lapse 0 ms., most likely because repository is not connected (UNABLE_TO_CONNECT).
at com.dynatrace.diagnostics.repository.exceptiontranslators.DefaultExceptionTranslator.a(SourceFile:65)
at com.dynatrace.diagnostics.repository.exceptiontranslators.DefaultExceptionTranslator.translateException(SourceFile:60)
at com.dynatrace.diagnostics.repository.statement.DefaultStatementExecutor.writeOnTransactionalConnection(SourceFile:95)
at com.dynatrace.diagnostics.repository.handlers.analysiscache.DefaultAnalysisCacheHandler.get(SourceFile:559)
... 15 more
Caused by: com.dynatrace.diagnostics.server.shared.exceptions.repository.NoConnectionException: IO Error: Got minus one from a read call, connect lapse 6983 ms., Authentication lapse 0 ms.
at com.dynatrace.diagnostics.repository.exceptiontranslators.OracleExceptionTranslator.doTranslateException(SourceFile:91)
... 18 more
2019-10-05 02:15:17 UTC INFO [DelegatingDataSource] An exception of class java.sql.SQLRecoverableException was thrown when trying to get a connection from the data source. Message : IO Error: Got minus one from a read call, connect lapse 7079 ms., Authentication lapse 0 ms., code: 17002, retry counter: 3
2019-10-05 02:15:17 UTC WARNING [AnalysisCacheEntry] Failed to age store LazyDatabaseCacheStore[key=AnalysisCacheKey [systemProfileId=, analyzerType=databaseWaitStates, splittings={databaseAgent=UAT_DFC_IQ_ALSCOMARCHIVE_MSSQL}], delegate=DatabaseCacheStore[key=AnalysisCacheKey [systemProfileId=, analyzerType=databaseWaitStates, splittings={databaseAgent=UAT_DFC_IQ_ALSCOMARCHIVE_MSSQL}],analysisCacheKeyId=-941380567]] [log message will be suppressed for 1 hours]: com.dynatrace.diagnostics.common.logging.ThrottlingLogger logViaJdk:156
com.dynatrace.diagnostics.repository.handlers.analysiscache.api.AnalysisCacheHandlerException: No connection to the PWH. State is UNABLE_TO_CONNECT
at com.dynatrace.diagnostics.server.analysiscache.store.DatabaseCacheStoreManager.d(SourceFile:114)
at com.dynatrace.diagnostics.server.analysiscache.store.DatabaseCacheStoreManager.mergeData(SourceFile:233)
at com.dynatrace.diagnostics.server.analysiscache.store.DatabaseCacheStore.mergeChunks(SourceFile:123)
at com.dynatrace.diagnostics.server.analysiscache.store.LazyDatabaseCacheStore.mergeChunks(SourceFile:102)
at com.dynatrace.diagnostics.server.analysiscache.AnalysisCacheEntry.age(SourceFile:353)
at com.dynatrace.diagnostics.server.analysiscache.AnalysisCache.age(SourceFile:478)
at com.dynatrace.diagnostics.server.analysiscache.AnalysisCacheFacade$4.run(SourceFile:346)
at com.dynatrace.diagnostics.server.ScheduledThreadPoolExecutorWithExceptionListener$2.run(SourceFile:82)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

performance warehouse
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

1 Answer

Sebastian K. avatar image
Sebastian K. answered ·

There can be plenty of reasons, network issue, problem with resources or connection pools on DB etc. Did you try some investigation in this area?

Sebastian

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.