Fixing "Maximum call stack size exceeded" in async Node.js code

I've happily used the async module for Node.js for years to simplify asynchronous operations over arrays. This is a difficulty in the Node.js paradigm, since the "normal" way to process an array is with a simple for loop. But if the operation is asynchronous the for loop has no way of knowing when to iterate to the next item, or even when processing is finished. Instead the async.eachSeries function does the trick, because your code tells async.eachSeries when to go to the next item, when there's an error, and it knows when the last item is processed, etc. It's been great.

But I just came across this:


RangeError: Maximum call stack size exceeded

The line of code in the stack trace pointed to a "next()" call that would iterate to the next entry in an async.eachSeries loop.

After some pondering and reading it seems that under the covers async.eachSeries creates a new level of stack recursion each time next() is called. Whoops. With a large enough array that will inevitably exceed the call stack size.


async.eachSeries(data, (datum, next) =&gt; {
        try { var d = new Date(datum["Timestamp"]).toISOString(); } catch (e) { d = undefined; }
        if (d) {
            db.run(`INSERT INTO ${tblnm} ( timestamp, json ) VALUES ( ?, ? )`, [
                d,
                JSON.stringify(datum)
            ],
            err =&gt; {
                if (err) next(err);
                else next();
            });
        } else next();
    },
    err =&gt; {
        if (err) reject(err);
        else resolve(db);
});

That's the code ...

The solution was simple ... replace the two next() calls with this: setTimeout(() => { next(); });

What this codes is reset the call stack before calling next(). Therefore the call stack will not be exceeded.

Another possibility would be something like this:


return Promise.all(data.map(datum =&gt; {
   return new Promise((resolve, reject) =&gt; {
       // perform async operation on datum
   });
})

But I did not try this. With async.eachSeries it is guaranteed to do one operation at a time, which isn't guaranteed with Promise.all. Hence, isn't there a risk of blowing up some other limit?

Promise.all is to be given an array of Promise's which it executes and finishes when all Promises have executed. It is another way of neatly executing an operation over an array. The data.map operation creates that array of Promise objects. In theory it would work, but we want to do this without blowing limits.

«««««« Implementing rsync or sftp in Node.js to synchronize files? Is Node.js / Express scalable? How to grow Node.js app deployment to handle more traffic? »»»»»»