question

Romain B. avatar image
Romain B. asked ·

Alerts not reaching Splunk with version 2.3.0 of the Splunk Plugin

Hey @Michael V. (I hope you are the right Michael ;-) ).

I spotted two issues with the latest version of the Splunk plugin (2.3.0).

1) There is still a reference to 1.3.1 in the path for log4j in runFlume.py at line 39, which breaks logging for flume itself:

It should be: log4j = os.path.join(appdir,"bin", "apache-flume-1.6.0-bin", "conf", "log4j.properties")

and not

log4j = os.path.join(appdir,"bin", "apache-flume-1.3.1-bin", "conf", "log4j.properties")

2) Secondly, every time an alert is being sent to Splunk using the latest version of the Dynatrace Splunk Alert Plugin, flume logs the following error and the alert isn't logged (and therefore not indexed):

12 May 2017 10:16:17,936 WARN [650949596@qtp-1653254447-0] (org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost:242) - Received bad request from client. org.apache.flume.source.http.HTTPBadRequestException: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.dynatrace.diagnostics.btexport.flume.BtExportHandler.getEvents(Unknown Source) at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:240) at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) at javax.servlet.http.HttpServlet.service(HttpServlet.java:814) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) at com.google.protobuf.CodedInputStream.readGroup(CodedInputStream.java:241) at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:488) at com.google.protobuf.GeneratedMessage.parseUnknownField(GeneratedMessage.java:193) at com.dynatrace.diagnostics.core.realtime.export.BtExport$BusinessTransactions.<init>(Unknown Source) at com.dynatrace.diagnostics.core.realtime.export.BtExport$BusinessTransactions.<init>(Unknown Source) at com.dynatrace.diagnostics.core.realtime.export.BtExport$BusinessTransactions$1.parsePartialFrom(Unknown Source) at com.dynatrace.diagnostics.core.realtime.export.BtExport$BusinessTransactions$1.parsePartialFrom(Unknown Source) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) at com.dynatrace.diagnostics.core.realtime.export.BtExport$BusinessTransactions.parseFrom(Unknown Source)

alerting6.5pluginssplunk
2 comments
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Same issues Alerts are not coming to Splunk with 2.3.0 ...... @Shane K @Ari P.

So I have downloaded the dynatrace 2.3.0 app and configured the same way as the old version and it sends the pp data but not alerts data any solutions or who can help please suggest guys

0 Likes 0 · ·

Hello, have you solved this issue? We have the same situation. Updating the Splunk App to 2.4.0 doesn't solve the issue. No alerts are being reported to flume port and we have no logging both on AppMon and Splunk sides. We have no clues. Any help? Maybe from developer? Thank you

0 Likes 0 · ·
ken l. avatar image
ken l. answered ·

Dynatrace, I understand that the plugin is not supported, however, there does not seem to be a supported method for sending alerts from APPMON to splunk. It is significant that the events be sent at the time the incident occurs. REST API or Dashboard retrieves historical data and causes multiple events per alert.
Can someone reach out to the developer of the plugin and see if he can provided assistance, or suggest a supported method.

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

ken l. avatar image
ken l. answered ·

was anyone able to resolve this or work around the issue

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Shane K. avatar image
Shane K. answered ·

I am not aware of any updates... @Michael V. ?

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Puneet K. avatar image
Puneet K. answered ·

Can you please get any updates on the above error @Shane K.

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Puneet K. avatar image
Puneet K. answered ·

any updates on this ?

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.