• Forums
    • Public Forums
      • Community Connect
      • Dynatrace
        • Dynatrace Open Q&A
      • Application Monitoring & UEM
        • AppMon & UEM Open Q&A
      • Network Application Monitoring
        • NAM Open Q&A
  • Home
  • Public Forums
  • Network Application Monitoring
  • NAM Open Q&A
avatar image
Question by Munawar T. · Jul 29, 2015 at 08:37 AM ·

Central Analysis Server Alert Engine

Hi,

Currently in our email we receive a lot of notification from CAS Alert engine, example as provide below.

During Microsoft Patching in our environment seems this alert more coming, kindly advise how to pinpoint the problem. Thanks

- 0 of 7844 finished sessions were not decrypted due to no private key found, 0% of finished sessions not decrypted due to incompleted SSL handshake, 10.046% of finished sessions not decrypted ot partially decrypted .

Alert generated for the time interval from (Tue Jul 28 00:43:00 SGT 2015) to (Tue Jul 28 00:47:59 SGT 2015) based on 5 amdstats file(s), To see details for AMD go to Reference 

Comment

People who like this

0 Show 0
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

6 Replies

  • Sort: 
  • Most voted
  • Newest
  • Oldest
avatar image

Answer by Chris V. · Jul 31, 2015 at 05:21 AM

To track SSL decryption failures, you need to use rcon on the AMD to get the details to troubleshoot.

log on to the AMD, start

rcon

and use the command

show ssldecr status

The results detail the reasons for partial or failed decryption of sessions.

eg.

        SESSIONS:
             Total number of sessions=59545 (inProgress=7 Finished=59538)
             SSL protocol version breakdown per number of sessions:
                       supported versions: ssl3.0=0 tls1.0=55914 tls1.1=0 tls1.2=0
                       unsupported versions: ssl2.0=0 other versions=0 no version info=3631
             Long handshakes=5062 Short handshakes=49856 Compressed sessions=0 SessionTkt reused=0 SessionId reused=99618
             Finished sessions decrypted with no errors=54521 (91% of all finished sessions)
             Sessions in progress decrypting with no errors=7 (100% of all sessions in progress)
             Finished sessions decrypted partially=31 (0% of all finished sessions)
                       with a packet lost during payload data exchange=31
                       with a corrupted payload data packet=0
                       with decryption failed during payload data exchange=0
                       terminated by alert during payload data exchange=0
             Finished sessions not decrypted=4946 (8% of all finished sessions)
                       with no private key found=280 (new sessions=224 reused sessions=56)
                       with a packet lost during handshake=25 (new sessions=5 reused sessions=20)
                       with a corrupted handshake packet or incorrect handshake sequence=40 (new sessions=40 reused sessions=0)
                       with decryption broken during handshake=0 (new sessions=0 reused sessions=0)
                       with unsupported SSL version=0 (ssl2.0=0 otherVersions=0)
                       with unsupported SSL feature=996 (unsupported cipher=996 server key exchange=0)
                       with compression errors=0 (unsupported compression=0, cannot decompress control records=0 data records=0)
                       with RSA decryption failed=0, RSA invocations blocked=0 (new sessions=0 reused sessions=0)
                       reused sessions with no matching master session seen before=15
                       with incomplete SSL handshake=0 (new sessions=0 reused sessions=0)
                       closed without data=23
                       with invalid 'Hello' packet client=0, server=0
                       terminated by alert during handshake=0
                       reuse errors when PMS identified with session id=86, with session ticket=0
                       session not seen from the beginning=3567
                       with other errors=0
             Supplemental Data detected, server=0 client=0

From this dump, we can see about 8% of sessions aren't decrypted properly, there are numerous reasons:

no private key (280) - easy to fix, get the private key
packets lost - not easy to fix
corrupted/incomplete handshake - not easy to fix
unsupported SSL feature - unsupported cipher (easy to fix, change server configuration)
reused sessions with no master - not easy to fix
closed without data - no fix
reuse errors - not easy to fix
session not seen from beginning - no easy to fix

A lot of the above are probably due to packets being missed by the AMD, either at the start or mid session. Fix would involve SPANs aren't oversubscribed/dropping packets, AMD isn't overloaded etc.  I know this AMD suffers packet loss (it's a virtual machine in my test lab, and is low on resources), which is the root cause for the majority of the decryption failures.

unsupported cipher, fixable.  TLS introduces DH/DHE/ECE (Diffie Hellman, Elliptic Curve) cipher suites, these are not decryptable - it's mathematically impossible, not something that can be fixed.  Reconfigure the application servers to use a RSA based cipher suite.

 
Comment
Carl M.

People who like this

1 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Munawar T. · Jul 31, 2015 at 07:28 AM 0
Share

Hi Chris,

Super explanation from the expert, appreciate your help.

I know that there is a documentation, but from your details information given can pinpoint the issue and we can use this as reference for further checking in our environments, determine what is the next possible steps and refer to cross check with DCRUM documentation.

Thank you,

Munawar

avatar image

Answer by Chris V. · Aug 11, 2015 at 03:38 AM

You'll have to figure out why the client/application is opening connections that aren't used.  That's not an AMD problem.

 

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Munawar T. · Aug 03, 2015 at 07:02 AM

Hi Chris,

Thanks for the comment, any further documentation that I can read for "closed without data=1442818" due the numbers is quite big or any recommended steps required to reduce the number.

As mentioned, We are trying to make our AMD as zero loss as possible.

Appreciate your guidance.

BR//Munawar

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Chris V. · Aug 02, 2015 at 11:34 PM

That's actually a pretty good result (1% failures).

The majority of your failures are due to sessions being closed before any data is transmitted.

closed without data=1442818

Without knowing more, I'd say this is some behavior of the client/application that establishes a connection needlessly.

I'd also check for lost packets, the other errors you're seeing are probably all related to packets the AMD missed seeing.

with a packet lost during payload data exchange=909
with incomplete SSL handshake=2814
reused sessions with no matching master session seen before=22107
session not seen from the beginning=1434

 

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Munawar T. · Jul 31, 2015 at 08:33 AM

After checking the SESSIONS in the AMD, here is the result. Any comment?

 

SSL DECRYPTION STATUS:
CONFIGURATION: Engine:openssl(thread) status:OK
Keys recognized=83 not recognized=0
SESSIONS:
Total number of sessions=89745613 (inProgress=391 Finished=89745222)
SSL protocol version breakdown per number of sessions:
supported versions: ssl3.0=0 tls1.0=88043404 tls1.1=147 tls1.2=257529
unsupported versions: ssl2.0=0 other versions=0 no version info=1444533
Long handshakes=13964648 Short handshakes=74336420 Compressed sessions=0 SessionTkt reused=0 SessionId reused=148650229
Finished sessions decrypted with no errors=88272072 (98% of all finished sessions)
Sessions in progress decrypting with no errors=391 (100% of all sessions in progress)
Finished sessions decrypted partially=913 (0% of all finished sessions)
with a packet lost during payload data exchange=909
with a corrupted payload data packet=3
with decryption failed during payload data exchange=0
terminated by alert during payload data exchange=1
Finished sessions not decrypted=1469448 (1% of all finished sessions)
with no private key found=0 (new sessions=0 reused sessions=0)
with a packet lost during handshake=119 (new sessions=119 reused sessions=0)
with a corrupted handshake packet or incorrect handshake sequence=25 (new sessions=25 reused sessions=0)
with decryption broken during handshake=37 (new sessions=37 reused sessions=0)
with unsupported SSL version=0 (ssl2.0=0 otherVersions=0)
with unsupported SSL feature=12 (unsupported cipher=12 server key exchange=0)
with compression errors=0 (unsupported compression=0, cannot decompress control records=0 data records=0)
reused sessions with no matching master session seen before=22107
with incomplete SSL handshake=2814 (new sessions=2811 reused sessions=3)
closed without data=1442818
with invalid 'Hello' packet client=0, server=0
terminated by alert during handshake=82
reuse errors when PMS identified with session id=20841, with session ticket=0
session not seen from the beginning=1434
with other errors=0
Supplemental Data detected, server=0 client=0
CERTIFICATES:
total server-certificate pairs=79
parsed properly=79 (matched=79 matching failed=0 not used=0)
parsing errors=0 (decode=0 extract=0 RSAerror=0)
RSA DECRYPTOR INTERNAL DIAGNOSTICS:
init/init errors (i=)13962416/0
finalize/finalize errors (f=)13962409/0
cancel/cancel errors (c=)3638/0
parallel curr/avg/max (p=)4294963665/70/4294967257
find key for cert init/fini/cancel/matched(f=)86/85/1/85
PMS CACHE INTERNAL DIAGNOSTICS:
entries added (a=)13964278 (asInitialized=13960867 asUninitialized=1161 withErrorCode=2250)
entries changed (c=)23650 (toInitialized=1542 toUninitialized=0 toError=22108)
entries deleted (d=)13959404
total entries in cache (n=)83180
SESSIONS ON HOLD DIAGNOSTICS:
total: 1437 max: 8 current: 0
PMS found: 168 not found: 1269

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Brett B. · Jul 30, 2015 at 07:43 PM

Hello Munawar,

 

Are you curious as to how to solve your 10.046% finished sessions not decrypted or partially decrypted or would you like to modify the alert and its threshold?

 

Regards,

Brett Barrett

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Munawar T. · Jul 31, 2015 at 02:11 AM 0
Share

Hello Brett,

Thank you for reply, In our dynatrace environments there is a principal "there is no data loss in the first place (it's apply to Dynatrace - DCRUM - Synthetic) unless you have details information to justified it". 

First, is it possible to solve the 10.046% of finished sessions not decrypted of partially decrypted? if yes, then kindly inform me what need to be done, what necessary thing need to be put in place (example: new SSL certs or new Private keys or anything).

Second, If not possible to solved the issue then we can thing of how to change the threshold, for this I'll do another research how to change it.

BR//Munawar

 

How to get started

First steps in the forum
Read Community User Guide
Best practices of using forum

NAM 2019 SP5 is available


Check the RHEL support added in the latest NAM service pack.

Learn more

LIVE WEBINAR

"Performance Clinic - Monitoring as a Self Service with Dynatrace"


JANUARY 15, 3:00 PM GMT / 10:00 AM ET

Register here

Follow this Question

Answers Answers and Comments

1 Person is following this question.

avatar image

Forum Tags

esm siebel Dynatrace Managed license nam probe wan citrix dna rest api configuration mq alerting NAM 2018 dashboard dcrumadvisory reports css nam universal decode database mobileapp RUM ads sap nam console scripting nam server sequence transactions nam 2019 upgrade
  • Forums
  • Public Forums
    • Community Connect
    • Dynatrace
      • Dynatrace Open Q&A
    • Application Monitoring & UEM
      • AppMon & UEM Open Q&A
    • Network Application Monitoring
      • NAM Open Q&A