Node js ECONNRESET

Go To StackoverFlow.com

226

I m running an Express js application with socket.io for a chat webapp and I get the following error randomly around 5 times during 24h. The node process is wrapped in forever and it restarts itself immediately.

Problem is that restarting express kicks my users out of their rooms and nobody wants that.

The web server is proxied by HAProxy. There are no socket stability issues, just using websockets and flashsockets transports. I cannot reproduce this on purpose.

This is the error with node v0.10.11:

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: read ECONNRESET     //alternatively it s a 'write'
    at errnoException (net.js:900:11)
    at TCP.onread (net.js:555:19)
error: Forever detected script exited with code: 8
error: Forever restarting script for 2 time

EDIT (2013-07-22)

Added both socket.io client error handler and the uncaught exception handler. Seems that this one catches the error:

process.on('uncaughtException', function (err) {
  console.error(err.stack);
  console.log("Node NOT Exiting...");
});

So I suspect it's not a socket.io issue but an http request to another server that I do or a mysql/redis connection. Problem is that the error stack doesn't help me identify my code issue. Here is the log output:

Error: read ECONNRESET
    at errnoException (net.js:900:11)
    at TCP.onread (net.js:555:19)

How do I know what causes this? How do I get more out of the error?

Ok, not very verbose but here s the stacktrace with "longjohn":

Exception caught: Error ECONNRESET
{ [Error: read ECONNRESET]
  code: 'ECONNRESET',
  errno: 'ECONNRESET',
  syscall: 'read',
  __cached_trace__:
   [ { receiver: [Object],
       fun: [Function: errnoException],
       pos: 22930 },
     { receiver: [Object], fun: [Function: onread], pos: 14545 },
     {},
     { receiver: [Object],
       fun: [Function: fireErrorCallbacks],
       pos: 11672 },
     { receiver: [Object], fun: [Function], pos: 12329 },
     { receiver: [Object], fun: [Function: onread], pos: 14536 } ],
  __previous__:
   { [Error]
     id: 1061835,
     location: 'fireErrorCallbacks (net.js:439)',
     __location__: 'process.nextTick',
     __previous__: null,
     __trace_count__: 1,
     __cached_trace__: [ [Object], [Object], [Object] ] } }

Here I serve the flash socket policy file:

net = require("net")
net.createServer( (socket) =>
  socket.write("<?xml version=\"1.0\"?>\n")
  socket.write("<!DOCTYPE cross-domain-policy SYSTEM \"http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd\">\n")
  socket.write("<cross-domain-policy>\n")
  socket.write("<allow-access-from domain=\"*\" to-ports=\"*\"/>\n")
  socket.write("</cross-domain-policy>\n")
  socket.end()
).listen(843)

Can this be the cause?

2013-06-21 23:49
by Samson
@GottZ maybe this can help (spoke to someone working within node js) https://gist.github.com/samsonradu/1b0c6feb438f5a53e30e. I ll deploy the socket.error handler today and let you know - Samson 2013-06-25 07:45
@Gottz the socket.error handles doesn t help, but process.on('uncaughtException') catches the error. Here is the console.log of the error: { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' - Samson 2013-07-03 20:37
ECONNRESET could be from network problem. As you know it is impossible to catch all the exceptions when testing. Some will show up on your production server. You will have to make your server robust. You can handle the session deletion by using Redis as storage. It makes your sessions persist even after your node server goes down - user568109 2013-07-08 05:32
Why is that related with session deletion? They are handled by Redis anyway - Samson 2013-07-08 07:28
This seams to me as you are getting a socket timeout from one of your connected clients. Try to add an 'error' listener to all your connections, not the process itself - Moss 2013-07-08 14:05
net.js does a lot of debuglog-ing when it's going to emit an error, you might get some mileage out of setting env.NODE_DEBUG="net" and then matching the output up against source locations - jjm 2013-07-09 14:49
@Moss define connected clients. The node server only listens to websocket connections and they have an error handler added - Samson 2013-07-10 10:49
You have at-least one TCP socket listening that does not have the handler set. So now it's time to check where that one is : - Moss 2013-07-10 12:34
just as note: in my case this happens with a childprocesses stdio and not a tcp socket or anything else. just raw stdio. will try that uncaughtException event now. thanks - GottZ 2013-07-11 17:35
Can you show the relevant code from your server where the error occurs. Or at least some minimal code to reproduce the error. It is difficult without looking at the code - user568109 2013-07-16 11:14
@user568109 here are some relevant parts (if you re fine with Coffeescript) https://gist.github.com/samsonradu/5585b0f4a101fee3cab - Samson 2013-07-16 11:26
https://github.com/joyent/node/releases/tag/v0.10.12 seeing a fix to socket writes. will upgrad - Samson 2013-07-16 16:52
Simply adding process.on('uncaughtException', function (err) { ... }) sorted this for me. It doesn't get to the root of the problem but the error is rare enough for me that this quick and dirty solution is fine for my use case - NoName 2017-04-06 04:45
Related: if anyone sees this happening on some cloud/PaaS offerings, it could be because the instances are throttling down (during quiet times) - opyate 2018-06-27 08:21


31

A simple tcp server I had for serving the flash policy file was causing this. I can now catch the error using a handler:

# serving the flash policy file
net = require("net")

net.createServer((socket) =>
  //just added
  socket.on("error", (err) =>
    console.log("Caught flash policy server socket error: ")
    console.log(err.stack)
  )

  socket.write("<?xml version=\"1.0\"?>\n")
  socket.write("<!DOCTYPE cross-domain-policy SYSTEM \"http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd\">\n")
  socket.write("<cross-domain-policy>\n")
  socket.write("<allow-access-from domain=\"*\" to-ports=\"*\"/>\n")
  socket.write("</cross-domain-policy>\n")
  socket.end()
).listen(843)
2013-07-23 08:38
by Samson
Is there anything wrong with the code? Should I have checked if the socket is writable before writing - Samson 2013-07-23 08:38
Doh, didn't see that you already found the solution before I posted pretty much the same thing :) As to your question though, even if you check that the socket is writable, it may not be when you write to it microseconds later and would still throw an error, so this is "the way" to be sure - Joachim Isaksson 2013-07-23 09:10
ok, and is there a safe way out if this? like socket.close() inside the error handler? because i think my CPU load is increasing after these errors (not sure - Samson 2013-07-23 09:49
I've always called socket.destroy() in the error handler to make sure. Sadly I can't find documentation whether it is required, but it does not emit an error to do so - Joachim Isaksson 2013-07-23 10:03
socket.destroy() saved my day, whatever it works !! Thank you - Firas Abd Alrahman 2016-10-29 18:18


209

You might have guessed it already: it's a connection error.

"ECONNRESET" means the other side of the TCP conversation abruptly closed its end of the connection. This is most probably due to one or more application protocol errors. You could look at the API server logs to see if it complains about something.

But since you are also looking for a way to check the error and potentially debug the problem, you should take a look at "How to debug a socket hang up error in NodeJS?" which was posted at stackoverflow in relation to an alike question.

Quick and dirty solution for development:

Use longjohn, you get long stack traces that will contain the async operations.

Clean and correct solution: Technically, in node, whenever you emit an 'error' event and no one listens to it, it will throw. To make it not throw, put a listener on it and handle it yourself. That way you can log the error with more information.

To have one listener for a group of calls you can use domains and also catch other errors on runtime. Make sure each async operation related to http(Server/Client) is in different domain context comparing to the other parts of the code, the domain will automatically listen to the error events and will propagate it to it's own handler. So you only listen to that handler and get the error data. You also get more information for free.

EDIT (2013-07-22)

As I wrote above:

"ECONNRESET" means the other side of the TCP conversation abruptly closed its end of the connection. This is most probably due to one or more application protocol errors. You could look at the API server logs to see if it complains about something.

What could also be the case: at random times, the other side is overloaded and simply kills the connection as a result. If that's the case, depends on what you're connecting to exactly…

But one thing's for sure: you indeed have a read error on your TCP connection which causes the exception. You can see that by looking at the error code you posted in your edit, which confirms it.

2013-07-14 08:53
by e-sushi
It doesn't have to mean 'abruptly closed'. It usually results from writing to a connection which the peer had already closed normally. That will cause it to issue an RST - user207421 2014-04-06 23:50
@EJP There was a good reason why I wrote “abruptly”. The error (not warning) states the connection was reset by peer. An existing connection was forcibly closed by the remote peer. A forced close is abrupt since unexpected! (This normally results if peer application on remote machine is suddenly stopped, machine is rebooted, or peer application used a "hard close" on the remote socket. This error may also result if a connection was broken due to "keep-alive" activity detecting a failure while one or more operations are in progress… these operations and subsequent operations will fail. - e-sushi 2014-04-07 09:12
I get this error thrown when I batch send around 100 API calls near concurrently from the browser (Chrome) for testing. I imagine that Chrome must then become overloaded and kill some of the connections...

@Samson - what is wrong with processing each request in its own domain and catching domain errors without restarting the server - supershnee 2014-07-10 08:12

@supershnee You should almost always restart your server after an uncaught exception since your data, application, and node.js itself is in an unknown state. Continuing after an exception puts your data a risk. If you want to find out more, check out Node's docs on process or Node's docs on domains - c1moore 2015-08-31 00:10


24

I had a similar problem where apps started erroring out after an upgrade of Node. I believe this can be traced back to Node release v0.9.10 this item:

  • net: don't suppress ECONNRESET (Ben Noordhuis)

Previous versions wouldn't error out on interruptions from the client. A break in the connection from the client throws the error ECONNRESET in Node. I believe this is intended functionality for Node, so the fix (at least for me) was to handle the error, which I believe you did in unCaught exceptions. Although I handle it in the net.socket handler.

You can demonstrate this:

Make a simple socket server and get Node v0.9.9 and v0.9.10.

require('net')
    .createServer( function(socket) 
    {
           // no nothing
    })
    .listen(21, function()
     {
           console.log('Socket ON')
    })

Start it up using v0.9.9 and then attempt to FTP to this server. I'm using FTP and port 21 only because I'm on Windows and have an FTP client, but no telnet client handy.

Then from the client side, just break the connection. (I'm just doing Ctrl-C)

You should see NO ERROR when using Node v0.9.9, and ERROR when using Node v.0.9.10 and up.

In production, I use v.0.10. something and it still gives the error. Again, I think this is intended and the solution is to handle the error in your code.

2013-07-22 22:12
by John Williams
Thanks, I nailed it myself! It s important not to let errors propagate to uncaughtException because it renders the whole app unstable. E.g. after catching around 10 ECONNRESET errors the server sometimes became unresponsive (just froze and didn t handle any connections - Samson 2013-07-23 08:57
Also knew about the node version change that didn t suppress the error any more, but seeing so many issues showing up and being solved each version I d rather go for the latest one. I m using V0.10.13 now bt - Samson 2013-07-23 08:59


14

I was facing the same issue but I mitigated it by placing:

server.timeout = 0;

before server.listen. server is an HTTP server here. The default timeout is 2 minutes as per the API documentation.

2014-10-28 05:07
by Ashish Kaila
This isn't a solution but rather a quickfix which will breaks things without throwing an error - Nishant Ghodke 2018-02-12 06:52


12

Had the same problem today. After some research i found a very useful --abort-on-uncaught-exception node.js option. Not only it provides much more verbose and useful error stack trace, but also saves core file on application crash allowing further debug.

2015-06-29 18:40
by stainful
weird that a new answer to this old question should pop up as I'm looking -- but this is great, thank - Semicolon 2015-06-29 18:41


7

Yes, your serving of the policy file can definitely cause the crash.

To repeat, just add a delay to your code:

net.createServer( function(socket) 
{
  for(i=0; i<1000000000; i++);
  socket.write("<?xml version=\"1.0\"?>\n")
…

…and use telnet to connect to the port. If you disconnect telnet before the delay has expired, you'll get a crash (uncaught exception) when socket.write throws an error.

To avoid the crash here, just add an error handler before reading/writing the socket:

net.createServer( function(socket) 
{
  for(i=0; i<1000000000; i++);
  socket.on('error', function() { console.log("error"); });
  socket.write("<?xml version=\"1.0\"?>\n")

When you try the above disconnect, you'll just get a log message instead of a crash.

And when you're done, remember to remove the delay.

2013-07-23 09:07
by Joachim Isaksson


5

Another possible case (but rare) could be if you have server to server communications and have set server.maxConnections to a very low value.

In node's core lib net.js it will call clientHandle.close() which will also cause error ECONNRESET:

if (self.maxConnections && self._connections >= self.maxConnections) {
  clientHandle.close(); // causes ECONNRESET on the other end
  return;
}
2016-10-07 14:32
by happy_marmoset


2

I also get ECONNRESET error during my development, the way I solve it is by not using nodemon to start my server, just use "node server.js" to start my server fixed my problem.

It's weird, but it worked for me, now I never see the ECONNRESET error again.

2018-04-26 16:18
by Andrew Lam


1

I solved the problem by simply connecting to a different network. That is one of the possible problems.

As discussed above, ECONNRESET means that the TCP conversation abruptly closed its end of the connection.

Your internet connection might be blocking you from connecting to some servers. In my case, I was trying to connect to mLab ( cloud database service that hosts MongoDB databases). And my ISP is blocking it.

2018-02-10 11:27
by Yousef
This one worked for me, my code which was working fine few hours back suddenly stopped working , turns out, the network change caused the proble - Aklank Jain 2018-04-12 04:39


1

I had this Error too and was able to solve it after days of debugging and analysis:

my solution

For me VirtualBox (for Docker) was the Problem. I had Port Forwarding configured on my VM and the error only occured on the forwarded port.

general conclusions

The following observations may save you days of work I had to invest:

  • For me the problem only occurred on connections from localhost to localhost on one port. -> check changing any of these constants solves the problem.
  • For me the problem only occurred on my machine -> let someone else try it.
  • For me the problem only occurred after a while and couldn't be reproduced reliably
  • My Problem couldn't be inspected with any of nodes or expresses (debug-)tools. -> don't waste time on this

-> figure out if something is messing around with your network (-settings), like VMs, Firewalls etc., this is probably the cause of the problem.

2018-05-04 16:02
by Waog


0

Try adding these options to socket.io:

const options = { transports: ['websocket'], pingTimeout: 3000, pingInterval: 5000 };

I hope this will help you !

2018-04-19 17:40
by sol404


0

I had resolved this problem by:

  • Turning off my wifi/ethernet connection and turn on.
  • I typed: npm update in terminal to update npm.
  • I tried to log out from the session and log in again

After that I tried the same npm command and the good thing was it worked out. I wasn't sure it is that simple.

I am using CENTOS 7

2019-02-25 11:46
by muhammad tayyab
Ads