• Forums
    • Public Forums
      • Community Connect
      • Dynatrace
        • Dynatrace Open Q&A
      • Application Monitoring & UEM
        • AppMon & UEM Open Q&A
      • Network Application Monitoring
        • NAM Open Q&A
  • Home /
  • Public Forums /
  • Application Monitoring & UEM /
  • AppMon & UEM Open Q&A /
avatar image
Question by Pranil C. · May 05, 2015 at 04:00 PM ·

Flume - Hadoop errror

Hi,

I am trying to setup dynatrace big data plugin for hadoop using flume.I have configured flume and hadoop.

However,after running flume ,i am getting following error :

5 May 2015 15:03:49,963 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating instance of sink: VisitSink, type: hdfs
05 May 2015 15:03:49,967 ERROR [conf-file-poller-0] (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:207) - Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:214)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 14 more

 

Can Somebody help please?

Comment

People who like this

0 Show 0
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

2 Replies

  • Sort: 
  • Most voted
  • Newest
  • Oldest
avatar image

Answer by Jeff F. · Jun 02, 2015 at 06:44 PM

It seems like you are missing *hadoop-core.jar* in the class path of flume. (see here or here) Make sure your HADOOP_HOME variable points to where your HADOOP installation is.

Can you provide some more details on the environment on where you are running flume?

 

-Jeff

 

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Pranil C. · Jun 03, 2015 at 12:00 PM 0
Share

Hi Jeff,

The hadoop core jar has been placed in flume lib folder and the slf4j jars too :

01 Jun 2015 15:56:11,368 ERROR [conf-file-poller-0] (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:211)  - Unhandled error
java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;Ljava/lang/Throwable;)V
 at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:120)
 at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:220)
 at org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:54)
 at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:106)
 at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:208)
 at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:529)
 at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:247)
 at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
 at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
 at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
 at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
 at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
 at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
01 Jun 2015 15:56:42,692 INFO  [node-shutdownHook] (org.apache.flume.node.FlumeNode.stop:67)  - Flume node stopping - QKDSK305
01 Jun 2015 15:56:42,692 INFO  [node-shutdownHook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78)  - Stopping lifecycle supervisor 8
01 Jun 2015 15:56:42,693 INFO  [node-shutdownHook] (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215)  - Node manager stopping
01 Jun 2015 15:56:42,693 INFO  [node-shutdownHook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78)  - Stopping lifecycle supervisor 8
01 Jun 2015 15:56:42,693 INFO  [node-shutdownHook] (org.apache.flume.conf.file.AbstractFileConfigurationProvider.stop:91)  - Configuration provider stopping

 

Kindly help.

avatar image

Answer by Jeff F. · Jun 03, 2015 at 08:44 PM

What version of slf4j are you using?

It look like this method signature changed between SLF4J 1.5x and SLF4J 1.6.x.

Can you provide some more details on the environment on where you are running flume including the version of the different components? I.e. Flume, Hadoop, SLF4J, etc

-Jeff

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Pranil C. · Jun 04, 2015 at 02:15 PM 0
Share

Hi Jeff,

Below are the versions i am using :

Dynatrace - 6.1 running on Windows 7 64 bit

Apache flume - 1.3.1 running on Centos 6.6

Hadoop - 2.3.0 (build for 64 bit) running on Windows 7 64 bit

hadoop-common-2.3.0.jar

hadoop-core-1.0.4.jar

hadoop-hdfs-2.3.0.jar

hadoop-mapreduce-client-core-2.3.0.jar

commons-configuration-1.6.jar

slf4j-api-1.6.1.jar

slf4j-log4j12-1.6.1.jar

 

Thanks,

Pranil

 

How to get started

First steps in the forum
Read Community User Guide
Best practices of using forum

NAM 2019 SP5 is available


Check the RHEL support added in the latest NAM service pack.

Learn more

LIVE WEBINAR

"Performance Clinic - Monitoring as a Self Service with Dynatrace"


JANUARY 15, 3:00 PM GMT / 10:00 AM ET

Register here

Follow this Question

Answers Answers and Comments

2 People are following this question.

avatar image avatar image

Forum Tags

dotnet mobile monitoring load iis 6.5 kubernetes mainframe rest api dashboard framework 7.0 appmon 7 health monitoring adk log monitoring services auto-detection uem webserver test automation license web performance monitoring ios nam probe collector migration mq web services knowledge sharing reports window java hybris javascript appmon sensors good to know extensions search 6.3+ server documentation easytravel web dashboard kibana system profile purelytics docker splunk 6.1 process groups account 7.2 rest dynatrace saas spa guardian appmon administration production user actions postgresql upgrade oneagent measures security Dynatrace Managed transactionflow technologies diagnostics user session monitoring unique users continuous delivery sharing configuration alerting NGINX splitting business transaction client 6.3 installation database scheduler apache mobileapp RUM php dashlet azure purepath agent 7.1 appmonsaas messagebroker nodejs 6.2 android sensor performance warehouse
  • Forums
  • Public Forums
    • Community Connect
    • Dynatrace
      • Dynatrace Open Q&A
    • Application Monitoring & UEM
      • AppMon & UEM Open Q&A
    • Network Application Monitoring
      • NAM Open Q&A