Central Analysis Server Alert Engine

To track SSL decryption failures, you need to use rcon on the AMD to get the details to troubleshoot.

log on to the AMD, start

rcon

and use the command

show ssldecr status

The results detail the reasons for partial or failed decryption of sessions.

eg.

        SESSIONS:
             Total number of sessions=59545 (inProgress=7 Finished=59538)
             SSL protocol version breakdown per number of sessions:
                       supported versions: ssl3.0=0 tls1.0=55914 tls1.1=0 tls1.2=0
                       unsupported versions: ssl2.0=0 other versions=0 no version info=3631
             Long handshakes=5062 Short handshakes=49856 Compressed sessions=0 SessionTkt reused=0 SessionId reused=99618
             Finished sessions decrypted with no errors=54521 (91% of all finished sessions)
             Sessions in progress decrypting with no errors=7 (100% of all sessions in progress)
             Finished sessions decrypted partially=31 (0% of all finished sessions)
                       with a packet lost during payload data exchange=31
                       with a corrupted payload data packet=0
                       with decryption failed during payload data exchange=0
                       terminated by alert during payload data exchange=0
             Finished sessions not decrypted=4946 (8% of all finished sessions)
                       with no private key found=280 (new sessions=224 reused sessions=56)
                       with a packet lost during handshake=25 (new sessions=5 reused sessions=20)
                       with a corrupted handshake packet or incorrect handshake sequence=40 (new sessions=40 reused sessions=0)
                       with decryption broken during handshake=0 (new sessions=0 reused sessions=0)
                       with unsupported SSL version=0 (ssl2.0=0 otherVersions=0)
                       with unsupported SSL feature=996 (unsupported cipher=996 server key exchange=0)
                       with compression errors=0 (unsupported compression=0, cannot decompress control records=0 data records=0)
                       with RSA decryption failed=0, RSA invocations blocked=0 (new sessions=0 reused sessions=0)
                       reused sessions with no matching master session seen before=15
                       with incomplete SSL handshake=0 (new sessions=0 reused sessions=0)
                       closed without data=23
                       with invalid 'Hello' packet client=0, server=0
                       terminated by alert during handshake=0
                       reuse errors when PMS identified with session id=86, with session ticket=0
                       session not seen from the beginning=3567
                       with other errors=0
             Supplemental Data detected, server=0 client=0

From this dump, we can see about 8% of sessions aren't decrypted properly, there are numerous reasons:

no private key (280) - easy to fix, get the private key
packets lost - not easy to fix
corrupted/incomplete handshake - not easy to fix
unsupported SSL feature - unsupported cipher (easy to fix, change server configuration)
reused sessions with no master - not easy to fix
closed without data - no fix
reuse errors - not easy to fix
session not seen from beginning - no easy to fix

A lot of the above are probably due to packets being missed by the AMD, either at the start or mid session. Fix would involve SPANs aren't oversubscribed/dropping packets, AMD isn't overloaded etc.  I know this AMD suffers packet loss (it's a virtual machine in my test lab, and is low on resources), which is the root cause for the majority of the decryption failures.

unsupported cipher, fixable.  TLS introduces DH/DHE/ECE (Diffie Hellman, Elliptic Curve) cipher suites, these are not decryptable - it's mathematically impossible, not something that can be fixed.  Reconfigure the application servers to use a RSA based cipher suite.

You'll have to figure out why the client/application is opening connections that aren't used.  That's not an AMD problem.

Hi Chris,

Thanks for the comment, any further documentation that I can read for "closed without data=1442818" due the numbers is quite big or any recommended steps required to reduce the number.

As mentioned, We are trying to make our AMD as zero loss as possible.

Appreciate your guidance.

BR//Munawar

That's actually a pretty good result (1% failures).

The majority of your failures are due to sessions being closed before any data is transmitted.

closed without data=1442818

Without knowing more, I'd say this is some behavior of the client/application that establishes a connection needlessly.

I'd also check for lost packets, the other errors you're seeing are probably all related to packets the AMD missed seeing.

with a packet lost during payload data exchange=909
with incomplete SSL handshake=2814
reused sessions with no matching master session seen before=22107
session not seen from the beginning=1434

After checking the SESSIONS in the AMD, here is the result. Any comment?

SSL DECRYPTION STATUS:
CONFIGURATION: Engine:openssl(thread) status:OK
Keys recognized=83 not recognized=0
SESSIONS:
Total number of sessions=89745613 (inProgress=391 Finished=89745222)
SSL protocol version breakdown per number of sessions:
supported versions: ssl3.0=0 tls1.0=88043404 tls1.1=147 tls1.2=257529
unsupported versions: ssl2.0=0 other versions=0 no version info=1444533
Long handshakes=13964648 Short handshakes=74336420 Compressed sessions=0 SessionTkt reused=0 SessionId reused=148650229
Finished sessions decrypted with no errors=88272072 (98% of all finished sessions)
Sessions in progress decrypting with no errors=391 (100% of all sessions in progress)
Finished sessions decrypted partially=913 (0% of all finished sessions)
with a packet lost during payload data exchange=909
with a corrupted payload data packet=3
with decryption failed during payload data exchange=0
terminated by alert during payload data exchange=1
Finished sessions not decrypted=1469448 (1% of all finished sessions)
with no private key found=0 (new sessions=0 reused sessions=0)
with a packet lost during handshake=119 (new sessions=119 reused sessions=0)
with a corrupted handshake packet or incorrect handshake sequence=25 (new sessions=25 reused sessions=0)
with decryption broken during handshake=37 (new sessions=37 reused sessions=0)
with unsupported SSL version=0 (ssl2.0=0 otherVersions=0)
with unsupported SSL feature=12 (unsupported cipher=12 server key exchange=0)
with compression errors=0 (unsupported compression=0, cannot decompress control records=0 data records=0)
reused sessions with no matching master session seen before=22107
with incomplete SSL handshake=2814 (new sessions=2811 reused sessions=3)
closed without data=1442818
with invalid 'Hello' packet client=0, server=0
terminated by alert during handshake=82
reuse errors when PMS identified with session id=20841, with session ticket=0
session not seen from the beginning=1434
with other errors=0
Supplemental Data detected, server=0 client=0
CERTIFICATES:
total server-certificate pairs=79
parsed properly=79 (matched=79 matching failed=0 not used=0)
parsing errors=0 (decode=0 extract=0 RSAerror=0)
RSA DECRYPTOR INTERNAL DIAGNOSTICS:
init/init errors (i=)13962416/0
finalize/finalize errors (f=)13962409/0
cancel/cancel errors (c=)3638/0
parallel curr/avg/max (p=)4294963665/70/4294967257
find key for cert init/fini/cancel/matched(f=)86/85/1/85
PMS CACHE INTERNAL DIAGNOSTICS:
entries added (a=)13964278 (asInitialized=13960867 asUninitialized=1161 withErrorCode=2250)
entries changed (c=)23650 (toInitialized=1542 toUninitialized=0 toError=22108)
entries deleted (d=)13959404
total entries in cache (n=)83180
SESSIONS ON HOLD DIAGNOSTICS:
total: 1437 max: 8 current: 0
PMS found: 168 not found: 1269

Hello Munawar,

Are you curious as to how to solve your 10.046% finished sessions not decrypted or partially decrypted or would you like to modify the alert and its threshold?

Regards,

Brett Barrett