• Forums
    • Public Forums
      • Community Connect
      • Dynatrace
        • Dynatrace Open Q&A
      • Application Monitoring & UEM
        • AppMon & UEM Open Q&A
      • Network Application Monitoring
        • NAM Open Q&A
        • Enterprise Synthetic Monitoring
      • Synthetic Classic
        • Synthetic Classic Open Q&A
  • Home /
  • Public Forums /
  • Dynatrace /
  • Dynatrace Open Q&A /
avatar image
Question by Jan-Hendrik P. · Sep 08, 2017 at 06:15 AM · Dynatrace Managed oneagent plugins monitoring

Dynatrace HA Proxy

Good morning,

we implement a load balancer appliance running ha proxy and stunnel for SSL connections. We experianced, that a service with SSL (configure stunnel to accept HTTPS and forward it to HA Proxy and configure HA Proxy to send the request encrypted to the backend) runs well, but we do not see any incoming traffic at the backend, which has to come over the proxy.

Further more do we not see, that the SSL service, which is known in Dynatrace Managed, is not "proxied". It seems, that the flow or the transaction is broken bei the encryption.
Another service without SSL works fine in this case (so we do only HA Proxy configured with rewrite rule etc pp -> works fine in Dynatrace).

I installed the OneAgent on the appliance and configured Dynatrace to use the /stats page of the HA Proxy, we use the HA Proxy plugin.

How do we solve that?

Best regards,

Jan

Comment

People who like this

0 Show 0
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

11 Replies

  • Sort: 
  • Most voted
  • Newest
  • Oldest
avatar image

Answer by Jan-Hendrik P. · Sep 12, 2017 at 02:07 PM

Hi Krzysztof,

many many thanks for your statement.
The load balancer appliance we uses HA Proxy and in the configuration with the WUI the STunnel for SSL.
I will test it with SSL over HA Proxy to see, what I need to in the manual configuration (the manual configuration overwrites the wizards configuration - which is great *yeah*).

The point with the processes and CPU cores is interesting and I will check that. Many many thanks for your information.

Best regards,
Jan

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image Andrew M. · Sep 13, 2017 at 05:16 AM 0
Share

Hi Jan,

Just keep in mind that multi-processing will involve additional concerns, for example stick-table synchronisation.

Please note particularly the third paragraph from this chapter on "How HAProxy works": https://cbonte.github.io/haproxy-dconv/1.7/intro.html#chapter-3.2

Regards,

Andrew

avatar image

Answer by Krzysztof S. · Sep 12, 2017 at 09:57 AM

HAProxy is leveraging OpenSSL (same as stunnel), it's working very stable and configuration is fairly straightforward.

One thing to watch out for is performance - while HAProxy is very fast, it has single-threaded architecture optimized for high level of TCP traffic. SSL handling though is CPU bound, and especially full SSL handshake for new connections is quite costly. With one HAProxy process you could get into situation when one core is fully utilized and HAProxy cannot handle more incoming traffic while other cores are sitting idle.

So, depending on the profile of your service, you may need to test and tune the configuration. It is a good idea to adjust maxconn and maxsslrate params so that they match your hardware capabilities. For higher SSL loads you may need to switch haproxy into multi-process mode (with nbproc).

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image

Answer by Jan-Hendrik P. · Sep 12, 2017 at 09:32 AM

Hi Krzysztof,

a colleague of mine has experiences with STunnel, so we started with that. We use HA-Proxy 1.7, so I will give it a try. ,-)

What's your experience with HA-Proxy and SSL termination?

Best,
Jan

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image Andrew M. · Sep 13, 2017 at 05:04 AM 0
Share

Hi Jan,

As others have said, HA Proxy now has native SSL termination. STunnel was commonly used with HAProxy 1.4 and earlier as a configuration in order to overcome the limitations of those earlier versions of HAproxy. I'm not up to date on the development of STunnel, however HAProxy supports more recent HTTP/2 as a bonus.

I would suggest that your Statistics may not be configured correctly, as your screenshot shows 0 counts.

Additionally to the HTTP statistics page that you may use, I suggest you enable the UNIX socket statistics as per: https://cbonte.github.io/haproxy-dconv/1.7/management.html#chapter-9.3

The socket statistics is supported by Dynatrace plugin: https://help.dynatrace.com/monitoring-plugins/load-balancer-plugins/how-do-i-configure-haproxy-monitoring/

Regards,

Andrew

avatar image

Answer by Krzysztof S. · Sep 12, 2017 at 09:17 AM

out of curiosity - why using stunnel instead of HAProxy native SSL support? Are you constrained to use HAProxy pre 1.5 version or do you see other benefits of using stunnel?

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image

Answer by Jan-Hendrik P. · Sep 12, 2017 at 09:03 AM

Hi Krzysztof,

it is. It seems, that the STunnel for the SSL termination breaks the connections from the client to the backend. We noticed disturbance in the whole platform. ,-(

Well, Dynatrace has problems in showing the traffic right, which comes from the STunnel and is sent to the HA-Proxy (both on the same host an listening on the same IP but different port) . Dynatrace showed us, that the outgoing traffic is going to another backend as we expected, because we know the platform configuration. ,-)

It would be nice to get an STunnel plugin for Dynatrace Managed/OneAgent, but it could be difficult, hm?

Best,
Jan

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image

Answer by Krzysztof S. · Sep 12, 2017 at 08:57 AM

Good Morning,
thank you for the update. That was an interesting use case.

kind regards,
Kris

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image

Answer by Jan-Hendrik P. · Sep 12, 2017 at 08:43 AM

Good Morning,

we noticed, that our service will not work with the HA-Proxy and STunnel configuration we want. Perhaps it is a wrong idea.

So, thank you for you impressions and thought, I documented them here and we will use another approach.

Thank you for your help.

Best regards,

Jan

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image

Answer by Jan-Hendrik P. · Sep 08, 2017 at 10:45 AM

Hi,

thank you. I looked into the configuration of the HA-Proxy appliance (which comes from loadbalancer.org) and I see this

defaults
    mode http
    balance roundrobin
    timeout connect 4000
    timeout client 42000
    timeout server 43000
    log global
    option log-health-checks

and this

listen stats
    bind :7777 ssl crt $PATH/$to/$cert
    stats admin if TRUE
    stats enable
    stats hide-version
    stats uri /
    option httpclose
    stats auth $USER:$PASS

I think, your options are missing, right?
I look a bit deeper into it. Thanks.

Best,
Jan

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image

Answer by Krzysztof S. · Sep 08, 2017 at 10:39 AM

Hi,

in your HAproxy configuration, please make sure that all the relevant frontents/backends are included in the statistics, this can with "stats scope" directive in "defaults" section. Not sure if it would address the issue with viewing the transactions, but at least you'd be able to see the overall statistics and continue troubleshooting from there.

Example configuration:

defaults
     # statistics settings
     stats enable
     stats show-legends
     stats scope servers
     stats scope http-in
Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image

Answer by Jan-Hendrik P. · Sep 08, 2017 at 10:07 AM

Hi Krzysztof and other interested people. ,-)

I created some screen shots and published them here:
https://www.dropbox.com/sh/xy4tn8r950ib4ke/AACwx1x...

They are numbered in the right direction.

Well, starting from the point of the stunnel, we have a service here, terminating at port 5493 and working SSL. Ok, STunnel is a C program, we will not look into it with Dynatrace Managed. Check.

The STunnel runs on lb-n15-02, which is a loadbalancer, this is right.

At the second image one sees one service terminating at port 5493. That is right. I expect, that the service name stands here (because Dynatrace learned, that sageapp.questico.qintern.de:9493 exists and this is this point).

The third image shows the processes/services, which accept outgoing calls, which is the HA Proxy itselfs (LBINTERN), that is right and fapcas, which is not right, because, the STunnel does not know anything about fapcas, but fapcas is configured on the HA Proxy without SSL and works.

The fourth image shows the loadbalancer itselfs and shows great incoming calls from servers, which connect the loadbalancer on 5493. That is great.

The fifth image shows the outgoing traffic and the ssl backend is not seen (we configured HA Proxy to encrypt the traffic going to the backend).

So, our search/look terminates here.

Ok, starting to look from the backend point of view and sageapp-n15-01 is one of the backend servers. The IIS and the ASP.NET Application handle the incoming requests.
At the sixth image no one sees incoming traffic at all here, but this is not true, definitely not true.

The seventh image is slightly the same but we see outgoing traffic, its weird but ok with this application. .-/
The eighth image shows, that the IIS does not get incoming traffic. Hm ... and this ist also not true.

So, we do have some points here, what we like to fix. ,-)
I followed the documentation you pointed out to configure the OneAgent and the HA Plugin for Dynatrace and we do see here the the VIPs of the HA Proxy and their backend and have some statistics.
But now do we need to put the ends together here. How do we do that?

Thank you for your time.

Best regards,
Jan

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image Jan-Hendrik P. · Sep 08, 2017 at 10:17 AM 0
Share

Hi, well,

since the HA Proxy uses /stats to get metrices and connection information of the HA Proxy, I looked into it and I see nothing about the service I fix in the monitoring in Dynatrace.
The stats for sageapp.questico.qintern.de shows only zeros:

Perhaps this could be the reason, that OneAgent/Dynatrace do not see any traffic between the loadbalancer and the backend?

Best,
Jan

haproxy-stats.jpg (342.2 kB)
  • 1
  • 2
  • ›

Welcome to the
Dynatrace Community Forums

Check out the Community User Guide and First steps in the forum to learn how to get started.

Community Member of the Month
December 2019

Announcing Dynatrace's Community Member of the Month for December 2019, Enrico F.! Click here to read more!

Employee Member of the Month
December 2019

Announcing Dynatrace's Employee Member of the Month for December 2019, Silvia M.! Click here to read more!

Live webinar: Ensuring Digital Business Availability with Dynatrace

Learn how Dynatrace Real User Monitoring automatically detects errors that impact your end users caused by erroneous 3rd party or CDNs.
December 12, 4:00 pm CET / 10:00 am ET
Register here

Live webinar: Ensuring Digital Business Availability with Dynatrace

Learn how Dynatrace Real User Monitoring automatically detects errors that impact your end users caused by erroneous 3rd party or CDNs.
December 12, 4:00 pm CET / 10:00 am ET
Register here

Live webinar: Ensuring Digital Business Availability with Dynatrace

Learn how Dynatrace Real User Monitoring automatically detects errors that impact your end users caused by erroneous 3rd party or CDNs.
December 12, 4:00 pm CET / 10:00 am ET
Register here

Live webinar: Ensuring Digital Business Availability with Dynatrace

Learn how Dynatrace Real User Monitoring automatically detects errors that impact your end users caused by erroneous 3rd party or CDNs.
December 12, 4:00 pm CET / 10:00 am ET
Register here

Follow this Question

Answers Answers and Comments

21 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

AWS Integration with Dynatrace Managed 2 Answers

Support for OS Windows Mobile 5 2 Answers

Broker monitoring using Dynatrace Managed and appmon/dcrum 1 Answer

Weblogic 12 - longer start time after activating monitoring 3 Answers

IBM dashDB 1 Answer

Forum Tags

mobile monitoring dotnet iis chat kubernetes servicenow amazon web services mysql rest api cassandra dashboard oneagent sdk cmc application monitoring smartscape request attributes monitoring ui developer community user tagging log monitoring services ufo activegate auto-detection high five award webserver uem test automation license web performance monitoring ios migration management zones web services sso notifications host monitoring reports browser monitors java sap vmware maintenance window javascript appmon ai availability tipstricks automation extensions session replay permissions faq documentation problem detection http monitors easytravel network docker tags and metadata cloud foundry google cloud platform synthetic monitoring process groups account usability dynatrace saas gui paas openshift key user actions administration hosts user actions postgresql synthetic locations oneagent security Dynatrace Managed user management python technologies mongodb openstack user session monitoring continuous delivery citrix configuration alerting NGINX action naming linux nam installation database mission control apache mobileapp RUM php azure purepath davis plugins scripting aix nodejs android
  • Forums
  • Public Forums
    • Community Connect
    • Dynatrace
      • Dynatrace Open Q&A
    • Application Monitoring & UEM
      • AppMon & UEM Open Q&A
    • Network Application Monitoring
      • NAM Open Q&A
      • Enterprise Synthetic Monitoring
    • Synthetic Classic
      • Synthetic Classic Open Q&A