Flume - Hadoop errror
Hi,
I am trying to setup dynatrace big data plugin for hadoop using flume.I have configured flume and hadoop.
However,after running flume ,i am getting following error :
5 May 2015 15:03:49,963 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:70) - Creating instance of sink: VisitSink, type: hdfs
05 May 2015 15:03:49,967 ERROR [conf-file-poller-0] (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:207) - Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:214)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 14 more
Can Somebody help please?
It seems like you are missing hadoop-core.jar in the class path of flume. (see here or here) Make sure your HADOOP_HOME variable points to where your HADOOP installation is.
Can you provide some more details on the environment on where you are running flume?
-Jeff
Hi Jeff,
The hadoop core jar has been placed in flume lib folder and the slf4j jars too :
01 Jun 2015 15:56:11,368 ERROR [conf-file-poller-0] (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:211) - Unhandled error
java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;Ljava/lang/Throwable;)V
at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:120)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:220)
at org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:54)
at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:106)
at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:208)
at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:529)
at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:247)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
01 Jun 2015 15:56:42,692 INFO [node-shutdownHook] (org.apache.flume.node.FlumeNode.stop:67) - Flume node stopping - QKDSK305
01 Jun 2015 15:56:42,692 INFO [node-shutdownHook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78) - Stopping lifecycle supervisor 8
01 Jun 2015 15:56:42,693 INFO [node-shutdownHook] (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215) - Node manager stopping
01 Jun 2015 15:56:42,693 INFO [node-shutdownHook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78) - Stopping lifecycle supervisor 8
01 Jun 2015 15:56:42,693 INFO [node-shutdownHook] (org.apache.flume.conf.file.AbstractFileConfigurationProvider.stop:91) - Configuration provider stopping
Kindly help.
What version of slf4j are you using?
It look like this method signature changed between SLF4J 1.5x and SLF4J 1.6.x.
Can you provide some more details on the environment on where you are running flume including the version of the different components? I.e. Flume, Hadoop, SLF4J, etc
-Jeff
Hi Jeff,
Below are the versions i am using :
Dynatrace - 6.1 running on Windows 7 64 bit
Apache flume - 1.3.1 running on Centos 6.6
Hadoop - 2.3.0 (build for 64 bit) running on Windows 7 64 bit
hadoop-common-2.3.0.jar
hadoop-core-1.0.4.jar
hadoop-hdfs-2.3.0.jar
hadoop-mapreduce-client-core-2.3.0.jar
commons-configuration-1.6.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
Thanks,
Pranil
