Hadoop Sqoop Job Performance Monitoring
Hi,
This is regarding dynatrace and hadoop server monitoring. We have few sqoop jobs to pull the data from database to hadoop server and would like to know how to capture the performance metrics for these jobs exclusivly on the hadoop server. If possible, please guide us to any documentation available for configuring dynatrace agent to monitor at job level rather than providing for the entire server.
Regards,
Venkata
Hi. I am not an expert in Hadoop therefore the following information might not be all that you expect - but I hope it is a start
First - we do have special support for Hadoop MapReduce job performance analysis. Check out the following doc page: Hadoop MapReduce
Second - as you specifically interested in Sqoop job performance: Sqoop seems to be a Java-based tool which means you can instrument this tool with dynaTrace while Sqoop is importing data from your databases to hadoop server. Once you have the agent in place you can define your own custom sensors to get method level information for things that are going on witin Sqoop. The other approach would be to look for JMX Metrics. Once you have a dynaTrace Agent injected in a Java App you can query any JMX Meausures exposed by that App. That might even be the better approach assuming that Sqoop provides some JMX Metrics. Here is a link to the doc that explains how to Using Custom JMX and PMI Measures in Charts
Hope this helps as a start
