Pushing custom metrics to Dynatrace using OneAgent extension

[Mayank_Sh]

In my use case, I am retrieving metrics from endpoint and trying to push to Dynatrace SaaS using OneAgent SDK (ie.. plugin.py and plugin.json).

1. Plugin.json:

- As endpoint contains varied kind of metrics. So, I considered hardcoding each of the metrics under metrics sections along with their respective dimensions.

- I have not used config-ui and ui section.

- For all other fields, kindly have a look at attached screenshot.

Similarly, there are 40 more metrics.

2. Plugin.py:

I am retrieving metrics from endpoint using prometheus_client.parser :

from prometheus_client.parser import text_string_to_metric_families

For transformation of metrics in Dynatrace-supported format , I am using this part of code:

- "metric-server" I have kept as Process group for now. But, not sure whether it is right or wrong ?

For pushing metrics to Dynatrace , this is used:

 

Issues faced by :

- On uploading of SDK in Dynatrace SaaS, though dimensions are being created on Dynatrace with respective metric name but, data is not coming.

- Is there any alternative way from hardcoding metrics in plugin.json and keep it more dynamic?

[Mike_L]

Hi,

if I were you I’d definitely use extension framework 2.0 instead. It comes with a declarative framework for Prometheus so you don’t have to write any code.

You can find more details here:

https://www.dynatrace.com/support/help/extend-dynatrace/extensions20/data-sources/prometheus-extensi...

Mike

[Mayank_Sh]

Thanks for reply Mike,

We are looking into the extension framework 2.0. But, meanwhile, can you tell me any remedy associated with the above followed procedure to get metrics in Dynatrace ?

Also, to avoid hardcoding of plugin.json file.

Thanks.

[Julius_Loman · Accepted solution]

@Mayank_Sh  Extensions 1.0 require metrics specified in the plugin.json. The only way to get around it is to send the metrics using by calling the Metric API from the code instead of the SDK methods. The ingest data point Metric API
You can use the Python API client for that. So you will use the extension code just to fetch and process the metric, but the sending will be using the API. Not ideal, but it will work.

If it's only prometheus as the source, definitely use Extension 2.0 for that.

[Mayank_Sh]

Hi @Mike_L  and @Julius_Loman  , Thanks for reply.

We are trying Prometheus extension 2.0 now.

We are monitoring an application which has been deployed on Kubernetes cluster(GKE). We have oneagent running as a container inside the pods. As a part of extension 2.0 process , we are required to upload root certificate(ca.pem) in each Oneagent host. 

Challenge which are being faced:

1. We are not able to copy root certificate file inside the pod.
   • Command used by us is : kubectl cp ca.pem dynatrace-oneagent-bk6ns:/var/lib/dynatrace/oneagent/agent/config/certificates -n dynatrace
   • Error observed is :error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "25efd997d58778142223f246292fd80171dfc99aea1a3531c9ded878ddba9380": OCI runtime exec failed: exec failed: unable to start container process: exec: "tar": executable file not found in $PATH: unknown
2. Even though we resolve the first issue, if pod is getting restarted then, we will lose root certificate again because pod is not persisting data. What is the remedy for it ?

Also, we are not allowed to create custom OneAgent image.

[Mike_L]

The remedy from our services team would be to sign it with the Dynatrace cert, but that's not possible for you of course. Looking at the error there are some solutions described here: https://stackoverflow.com/questions/67087858/why-does-kubectl-cp-command-terminates-with-exit-code-1...