Dynatrace HA Proxy

[jan_palic]

Good morning,

we implement a load balancer appliance running ha proxy and stunnel for SSL connections. We experianced, that a service with SSL (configure stunnel to accept HTTPS and forward it to HA Proxy and configure HA Proxy to send the request encrypted to the backend) runs well, but we do not see any incoming traffic at the backend, which has to come over the proxy.

Further more do we not see, that the SSL service, which is known in Dynatrace Managed, is not "proxied". It seems, that the flow or the transaction is broken bei the encryption.
Another service without SSL works fine in this case (so we do only HA Proxy configured with rewrite rule etc pp -> works fine in Dynatrace).

I installed the OneAgent on the appliance and configured Dynatrace to use the /stats page of the HA Proxy, we use the HA Proxy plugin.

How do we solve that?

Best regards,

Jan

[krzysztof_szynt]

Hi,
do I understand correctly that you are trying to troubleshoot your service with Dynatrace but do not see the traffic you expected there? Can you provide a screenshot where it happens?

Regarding HAProxy plugin, it will help you monitor the level of traffic and diagnose issues with HAProxy, like overload, but it is based on statistics and won't contribute to transactional data.
You can find the HAProxy metrics in the context of HAProxy process - looking at "Further details"

More on HAProxy pluging configuration: https://help.dynatrace.com/monitoring-plugins/load...

As a side note, HAProxy does have native SSL support since version 1.5, have you considered using this instead of the stunnel?

kind regards,
Kris

[jan_palic]

Hi Krzysztof and other interested people. ,-)

I created some screen shots and published them here:
https://www.dropbox.com/sh/xy4tn8r950ib4ke/AACwx1x...

They are numbered in the right direction.

Well, starting from the point of the stunnel, we have a service here, terminating at port 5493 and working SSL. Ok, STunnel is a C program, we will not look into it with Dynatrace Managed. Check.

The STunnel runs on lb-n15-02, which is a loadbalancer, this is right.

At the second image one sees one service terminating at port 5493. That is right. I expect, that the service name stands here (because Dynatrace learned, that sageapp.questico.qintern.de:9493 exists and this is this point).

The third image shows the processes/services, which accept outgoing calls, which is the HA Proxy itselfs (LBINTERN), that is right and fapcas, which is not right, because, the STunnel does not know anything about fapcas, but fapcas is configured on the HA Proxy without SSL and works.

The fourth image shows the loadbalancer itselfs and shows great incoming calls from servers, which connect the loadbalancer on 5493. That is great.

The fifth image shows the outgoing traffic and the ssl backend is not seen (we configured HA Proxy to encrypt the traffic going to the backend).

So, our search/look terminates here.

Ok, starting to look from the backend point of view and sageapp-n15-01 is one of the backend servers. The IIS and the ASP.NET Application handle the incoming requests.
At the sixth image no one sees incoming traffic at all here, but this is not true, definitely not true.

The seventh image is slightly the same but we see outgoing traffic, its weird but ok with this application. .-/
The eighth image shows, that the IIS does not get incoming traffic. Hm ... and this ist also not true.

So, we do have some points here, what we like to fix. ,-)
I followed the documentation you pointed out to configure the OneAgent and the HA Plugin for Dynatrace and we do see here the the VIPs of the HA Proxy and their backend and have some statistics.
But now do we need to put the ends together here. How do we do that?

Thank you for your time.

Best regards,
Jan

[jan_palic]

Hi, well,

since the HA Proxy uses /stats to get metrices and connection information of the HA Proxy, I looked into it and I see nothing about the service I fix in the monitoring in Dynatrace.
The stats for sageapp.questico.qintern.de shows only zeros:

Perhaps this could be the reason, that OneAgent/Dynatrace do not see any traffic between the loadbalancer and the backend?

Best,
Jan

[krzysztof_szynt]

Hi,

in your HAproxy configuration, please make sure that all the relevant frontents/backends are included in the statistics, this can with "stats scope" directive in "defaults" section. Not sure if it would address the issue with viewing the transactions, but at least you'd be able to see the overall statistics and continue troubleshooting from there.

Example configuration:

defaults
     # statistics settings
     stats enable
     stats show-legends
     stats scope servers
     stats scope http-in

[jan_palic]

Hi,

thank you. I looked into the configuration of the HA-Proxy appliance (which comes from loadbalancer.org) and I see this

defaults
    mode http
    balance roundrobin
    timeout connect 4000
    timeout client 42000
    timeout server 43000
    log global
    option log-health-checks

and this

listen stats
    bind :7777 ssl crt $PATH/$to/$cert
    stats admin if TRUE
    stats enable
    stats hide-version
    stats uri /
    option httpclose
    stats auth $USER:$PASS

I think, your options are missing, right?
I look a bit deeper into it. Thanks.

Best,
Jan

[jan_palic]

Good Morning,

we noticed, that our service will not work with the HA-Proxy and STunnel configuration we want. Perhaps it is a wrong idea.

So, thank you for you impressions and thought, I documented them here and we will use another approach.

Thank you for your help.

Best regards,

Jan

[krzysztof_szynt]

Good Morning,
thank you for the update. That was an interesting use case.

kind regards,
Kris

[jan_palic]

Hi Krzysztof,

it is. It seems, that the STunnel for the SSL termination breaks the connections from the client to the backend. We noticed disturbance in the whole platform. ,-(

Well, Dynatrace has problems in showing the traffic right, which comes from the STunnel and is sent to the HA-Proxy (both on the same host an listening on the same IP but different port) . Dynatrace showed us, that the outgoing traffic is going to another backend as we expected, because we know the platform configuration. ,-)

It would be nice to get an STunnel plugin for Dynatrace Managed/OneAgent, but it could be difficult, hm?

Best,
Jan

[krzysztof_szynt]

out of curiosity - why using stunnel instead of HAProxy native SSL support? Are you constrained to use HAProxy pre 1.5 version or do you see other benefits of using stunnel?

[jan_palic]

Hi Krzysztof,

a colleague of mine has experiences with STunnel, so we started with that. We use HA-Proxy 1.7, so I will give it a try. ,-)

What's your experience with HA-Proxy and SSL termination?

Best,
Jan

[The_AM]

Hi Jan,

As others have said, HA Proxy now has native SSL termination. STunnel was commonly used with HAProxy 1.4 and earlier as a configuration in order to overcome the limitations of those earlier versions of HAproxy. I'm not up to date on the development of STunnel, however HAProxy supports more recent HTTP/2 as a bonus.

I would suggest that your Statistics may not be configured correctly, as your screenshot shows 0 counts.

Additionally to the HTTP statistics page that you may use, I suggest you enable the UNIX socket statistics as per: https://cbonte.github.io/haproxy-dconv/1.7/management.html#chapter-9.3

The socket statistics is supported by Dynatrace plugin: https://help.dynatrace.com/monitoring-plugins/load-balancer-plugins/how-do-i-configure-haproxy-monitoring/

Regards,

Andrew

[krzysztof_szynt · Accepted solution]

HAProxy is leveraging OpenSSL (same as stunnel), it's working very stable and configuration is fairly straightforward.

One thing to watch out for is performance - while HAProxy is very fast, it has single-threaded architecture optimized for high level of TCP traffic. SSL handling though is CPU bound, and especially full SSL handshake for new connections is quite costly. With one HAProxy process you could get into situation when one core is fully utilized and HAProxy cannot handle more incoming traffic while other cores are sitting idle.

So, depending on the profile of your service, you may need to test and tune the configuration. It is a good idea to adjust maxconn and maxsslrate params so that they match your hardware capabilities. For higher SSL loads you may need to switch haproxy into multi-process mode (with nbproc).

[jan_palic]

Hi Krzysztof,

many many thanks for your statement.
The load balancer appliance we uses HA Proxy and in the configuration with the WUI the STunnel for SSL.
I will test it with SSL over HA Proxy to see, what I need to in the manual configuration (the manual configuration overwrites the wizards configuration - which is great *yeah*).

The point with the processes and CPU cores is interesting and I will check that. Many many thanks for your information.

Best regards,
Jan

[The_AM]

Hi Jan,

Just keep in mind that multi-processing will involve additional concerns, for example stick-table synchronisation.

Please note particularly the third paragraph from this chapter on "How HAProxy works": https://cbonte.github.io/haproxy-dconv/1.7/intro.html#chapter-3.2

Regards,

Andrew