Java net socketexception broken pipe write failed

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

java.net.SocketException: Broken pipe (Write failed) when trying to post 150K+ json payload #50

java.net.SocketException: Broken pipe (Write failed) when trying to post 150K+ json payload #50

Comments

I’m having issues trying to POST «relatively big» json payloads (150K+) using the output http plugin.

I’ve tried to increase automatic_retries , connect_timeout, request_timeout, socket_timeout and decrease validate_after_inactivity properties with no success.

I’ve validated I can post the same payload via Postman or via curl within the logstash instance.
My current workaround is to save the event in a json file using the file output plugin and post it
via curl using the exec output plugin.
But obviously that’s not ideal.

I’ve enabled debugging on http-output plugin and some apache packages (wire and retryExec) (c.f stacktrace below)

It works fine w/ smaller payloads but seems to always fail for anything slightly bigger than 100K or so.
Is this a known issue or is there any limit on the payload size for this plugin?

• Version: logstash 5.1.1, http-output-plugin 3.1.1 running in a container
• Operating System: ubuntu 14.04.1
• config file:

input < beats < port =>5044 ssl => true ssl_certificate => "/etc/logstash/my.logstash.crt" ssl_key => "/etc/logstash/my.logstash.key" > > filter < if [type] == "inventory" < json < source =>"message" remove_field => ["message"] > > > output < if [type] == "inventory" < http < url =>"xxxxx/inventory" headers => < "Content-Type" =>"application/json" > http_method => "post" codec => "json" automatic_retries => 5 "http_inventory" socket_timeout => 1000 request_timeout => 1000 validate_after_inactivity => 0 retry_non_idempotent => true > > 

?21:27:54.670 [pool-186-thread-1] INFO org.apache.http.impl.execchain.RetryExec - Retrying request to ->https://xxxxx:443 ?21:27:54.706 [pool-186-thread-1] DEBUG org.apache.http.wire - http-outgoing-54 >> "POST /inventory HTTP/1.1[\r][\n]" > "Content-Type: application/json[\r][\n]" s21:27:54.706 [pool-186-thread-1] DEBUG org.apache.http.wire - http-outgoing-54 >> "Connection: Keep-Alive[\r][\n]" s21:27:54.706 [pool-186-thread-1] DEBUG org.apache.http.wire - http-outgoing-54 >> "Content-Length: 183475[\r][\n]" w21:27:54.706 [pool-186-thread-1] DEBUG org.apache.http.wire - http-outgoing-54 >> "Host: xxxxxxx[\r][\n]" x21:27:54.706 [pool-186-thread-1] DEBUG org.apache.http.wire - http-outgoing-54 >> "User-Agent: Manticore 0.6.0[\r][\n]" z21:27:54.706 [pool-186-thread-1] DEBUG org.apache.http.wire - http-outgoing-54 >> "Accept-Encoding: gzip,deflate[\r][\n]" ]21:27:54.706 [pool-186-thread-1] DEBUG org.apache.http.wire - http-outgoing-54 >> "[\r][\n]" 21:27:55.074 [pool-186-thread-1] DEBUG org.apache.http.wire - http-outgoing-54 >> "--- here there's the json payload that i can't really post --- ?21:27:55.103 [pool-186-thread-1] DEBUG org.apache.http.wire - http-outgoing-54 >> "[write] I/O error: Broken pipe (Write failed)" ?21:27:55.104 [pool-186-thread-1] INFO org.apache.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to ->https://xxxxx:443: Broken pipe (Write failed) m21:27:55.104 [pool-186-thread-1] DEBUG org.apache.http.impl.execchain.RetryExec - Broken pipe (Write failed) 5java.net.SocketException: Broken pipe (Write failed) K at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_111] X at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[?:1.8.0_111] R at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[?:1.8.0_111] T at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431) ~[?:1.8.0_111] N at sun.security.ssl.OutputRecord.write(OutputRecord.java:417) ~[?:1.8.0_111] ^ at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:876) ~[?:1.8.0_111] V at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:847) ~[?:1.8.0_111] T at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[?:1.8.0_111] s at org.apache.http.impl.conn.LoggingOutputStream.write(LoggingOutputStream.java:77) ~[httpclient-4.5.2.jar:4.5.2] ~ at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:126) ~[httpcore-4.4.4.jar:4.4.4] x at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:162) ~[httpcore-4.4.4.jar:4.4.4] | at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:115) ~[httpcore-4.4.4.jar:4.4.4] | at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:122) ~[httpcore-4.4.4.jar:4.4.4] c at org.apache.http.entity.StringEntity.writeTo(StringEntity.java:169) ~[httpcore-4.4.4.jar:4.4.4] ? at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:158) ~[httpcore-4.4.4.jar:4.4.4] n at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:162) ~[httpclient-4.5.2.jar:4.5.2] y at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:237) ~[httpcore-4.4.4.jar:4.4.4] s at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122) ~[httpcore-4.4.4.jar:4.4.4] q at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) ~[httpclient-4.5.2.jar:4.5.2] m at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] e at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) [httpclient-4.5.2.jar:4.5.2] l at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) [httpclient-4.5.2.jar:4.5.2] w at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) [httpclient-4.5.2.jar:4.5.2] v at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71) [httpclient-4.5.2.jar:4.5.2] w at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:220) [httpclient-4.5.2.jar:4.5.2] w at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164) [httpclient-4.5.2.jar:4.5.2] H at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) ~[. ] i at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111] D at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111] b at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:481) [. ] M at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:336) [. ] \ at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:68) [. ] T at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:236) [. ] ? at rubyjit.Manticore::Response$$call_28e9faa42def46f7b7c11e2eb043499f7b09a0a21173230247.chained_0_rescue_1$RUBY$SYNTHETIC__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:50) [. ] ? at rubyjit.Manticore::Response$$call_28e9faa42def46f7b7c11e2eb043499f7b09a0a21173230247.__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:49) [. ] ? at rubyjit.Manticore::Response$$call_28e9faa42def46f7b7c11e2eb043499f7b09a0a21173230247.__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb) [. ] ? at rubyjit.Manticore::Response$$call_28e9faa42def46f7b7c11e2eb043499f7b09a0a21173230247.__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb) [. ] V at org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:101) [. ] S at Manticore$$Response_875930374.call(Manticore$$Response_875930374.gen:13) [. ] K at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_111] b at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111] b at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111] 8 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111] 

The text was updated successfully, but these errors were encountered:

Источник