Kafka producer and consumer coupling

[dedopspy]

We have a microservice (let's say A) that produces messages into a Kafka topic. We also have another microservice (implemented as a Java Message Driven Bean) that consumes these messages and inserts them (modified) in a Redis database.

We use OneAgent SDK and we enrich the posted messages (from producer side) with a header named "dtdTraceTagInfo". On the consumer side, we have used OneAgentSDK to create an IncomingMessageReceiveTracer and an IncomingMessageProcessTracer, calling at the latter the "setDynatraceStringTag" method, passing as argument the retrieved dtdTraceTagInfo.

All these, in order to see a purepath that contains both the producer and the consumer of the message, but in vain. We only see 2 different set of traces (purpaths), one for the producer and one for the consumer, each one containing a different trace id.

Is there a way to couple Kafka producers and consumers, in order to see them both in a purepath?

Thanks in advance

[ben_wrightson]

This should work fine without having to use the OneAgent SDK (assuming you use Java on both the producer and the consumer side). Have you configured a custom service on the consumer side as described in https://www.dynatrace.com/support/help/how-to-use-dynatrace/services/custom-services/define-messagin... ?

[dedopspy]

Yes, we did, and we are able to see the service in the "Services" pane. 

I don't know how end to end tracing is implemented by Dynatrace in the case of Kafka, but I would assume that something is added to each message (again, I assume that this "something" is a special header and I would bet that it is the dtdTraceTageInfo) that enabled Dynatrace to correlate the sending with the receiving side. 

The problem is that when we don't use SDK for the manual insertion of the tag, it is not inserted as a header in the message (as I have found by using the kafka console consumer to see the contents of the messages).

Are we missing something here? Are we doing something wrong?

Thanks in advance

[ben_wrightson · Accepted solution]

In messaging, there might not be anything added to the message itself. If the message producer and consumer are both supported technologies, full-stack instrumented with the OneAgent and you've defined the custom service then this should work - I've seen this work fine before without having to use the OneAgent SDK at all.
I assume you've also turned on "Java Kafka" in Settings / Preferences / OneAgent features?
If that's not it then I'd recommend opening a support ticket.

[dedopspy]

You were right, we just needed some configuration changes for it to work out of the box. We found out that although it is needed a header in the message, this header is not the one described in the github (the one that the SDK produces) but, instead, the header that Dyantrace uses for web requests tracing (the X-dunatrace.... tag). Anyway, it worked and thanks for the tip.