question

Clederson C. avatar image
Clederson C. asked ·

Metrics with Kafka and Spark

Hey guys, how are you?


My squad have a project that we need to use Dynatrace as our main monitoring tool.

But we need to know if Dynatrace can track some metrics for Kafka and Spark.


Could you help us?


Down below there is a list of metrics we need to monitoring.


Kafka

Health Alert

consumed messages (Topics)

storage Kafka % --> Alert for 80% of storage


Spark

Health Alert

Total of Jobs

Number of bytes recorded

Number of bytes processed

Number of active's tasks

Number of failing Tasks alert

Total of Input Bytes (per second)

Total of Output Bytes (per second)

total of executors

waiting time ---> alert

storage ----> Alert 80%

% memory consume

% queue process

execution time ----> Alert (per Second) *

Average time of messages processed time ----> Alert*

alerting
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Sebastian K. avatar image
Sebastian K. answered ·

Remember that if some of metrics are missing out of the box but are available as jmx mbean, you can always create custom JMX plugin using built in JMX editor to expose those metrics. But you have to have license for custom metrics to do that.

https://www.dynatrace.com/news/blog/jmx-plugins-editor-now-available-eap/

Sebastian

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Clederson C. avatar image
Clederson C. answered ·

@Jonathan S. ,


Obrigado pelo seu tempo.

Na verdade, eu consegui essas páginas para identificar métricas, mas não encontrei nenhuma informação sobre as métricas que preciso.

Essas páginas têm todas as métricas que o Dynatrace pode suportar para kafka e nifi?

Cumprimentos,


Leandro R.

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Jonny S. avatar image
Jonny S. answered ·
Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Space Topics

mobile monitoring dotnet synthetic monitoring reports iis chat kubernetes servicenow amazon web services mysql mainframe rest api errors cassandra dashboard oneagent sdk cmc application monitoring openkit smartscape request attributes monitoring developer community user tagging log monitoring services ufo syntheticadvisory activegate ip addresses auto-detection high five award oracle hyperion webserver uem usql iib test automation license web performance monitoring ios news migration management zones index ibm mq web services notifications sso host monitoring knowledge sharing reports browser monitors java hybris sap vmware maintenance window user action naming javascript appmon ai synthetic classic availability tipstricks automation extensions diagnostic tools session replay permissions davis assistant faq documentation problem detection http monitors server easytravel apdex aws-quickstart network docker tags and metadata cloud foundry google cloud platform synthetic monitoring process groups account usability dynatrace saas gui paas openshift key user actions administration user actions postgresql synthetic locations oneagent security Dynatrace Managed user management custom python technologies mongodb openstack user session monitoring continuous delivery citrix configuration alerting NGINX action naming linux nam installation masking error reporting database mission control jmeter recorder apache mobileapp RUM php threshold azure purepath davis scripting agent aix nodejs android