OTEL (auto) ingest for K8s services

[r_weber]

Hello,
we are trying to get a vert.x java service traced with the support of OpenTelemetry.
The service runs in a K8s pod which is monitored already by Oneagent (dynakube).

The process should be very straight forward, just following these steps. Using oneagent ingestion.

• add OTEL agent to the java startup
• disable default exporters via env variables
  
  OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
  OTEL_EXPORTER=none

However when the application is start up it always still tries to perform the export to localhost:

SEVERE: Failed to export metrics. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:431884java.net.ConnectException: Failed to connect to localhost/127.0.0.1:431883	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297)82

When we try to set the exporter URL manually to directly export to the SaaS instance it seems to work (a connection attempt is made). But we cannot use that setup due to firewall restrictions.

Now I have a few questions:

In the above log the export tries to connect to localhost/127.0.0.1:431883... What port ist that? That doesn't seem to be a valid port range?
In that configuration (oneagent ingested) there shouldn't be any attempt to export traces anywhere, right? The oneagent within the process should pick up those traces, right?

Thanks!

[alexander_dt]

You are right, 431883 is way beyond port 65535. My guess for the network level would be, that value gets truncated and turns into 38667. Not that this would be a standard port for OpenTelemetry either, however.

As for the ingestion question, OneAgent used to disable that but doesn't do so any more as of version 1.259 - https://docs.dynatrace.com/docs/shortlink/opentelemetry-oneagent#java - so the error message itself is not surprising. That being said, I would usually assume setting "OTEL_EXPORTER" to "none" should do the trick and disable exporting on an OpenTelemetry level. Going by https://docs.dynatrace.com/docs/shortlink/otel-wt-java-auto#disable-default-trace-exporter, could you try to specifically set OTEL_TRACES_EXPORTER?

Other than that, that particular port number is very specific and seemingly does refer to the default port of OTLP via HTTP at port 4318, but I somewhat doubt any of the involed standard components would render a port invalid by appending "83" out of the blue. Could it be that you have this configured (in code or as environment variable) somewhere? The usual typo :). Could you grep the file system in question for "431883" or maybe just for "4318" and then check if something appends "83"?

Best,
Alexander

[r_weber]

It turned out that the documentation was wrong at the time of writing the above post.

Back then it mentioned to set OTEL_EXPORTER while it has to be OTEL_TRACES_EXPORTER.

I mentioned this to the doc team and they now fixed that.

[alexander_dt]

Right, that would be the global value. So there was still a traces specific value configured with that port?

[r_weber]

No, as per the recent OTEL documentation there is no such environment variable as OTEL_EXPORTER. Only OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER and OTEL_LOGS_EXPORTER.
Where that port is coming from we do not know, might be also som log/formatting issue maybe...