The ‘async’ library is probably one of the top ten packages you install when you start working on a NodeJS application and for good reason too. The library lets you write asynchronous code without all the boilerplate code out of the box.

I, personally, have a few favourites functions that are a must have in any code I work with - the waterfall, parallel, each, series, and the apply.

Recently, I tried out the queue function and I literally felt like a 10-year old kid again who just got a new video game. The function itself is pretty basic at what it does but the things you can do with it are endless.

I needed to download images from a remote URL and return it for use in another part of the application. The queue function seemed perfect for the job. I could push any incoming URL into the queue and it would get taken care of asychronously in the background.

To get started with the library, install the library and require it in your application -
npm install --save async
var async = require('async');

In your *.js file,

var q = async.queue(function (task, callback) {
  _someFunction(task, callback);  
}, 10);  

In the first line, we create an instance of the queue which takes in a worker function and the number of concurrent workers as the parameters.

We define the worker to call a private function of ours, which is the function that actually downloads the image for a remote URL (but that function is beyond the scope of this post). It can be any function though. The same parameters being passed in to the worker are the same parameters being passed into the private function of ours. The number of concurrent workers, is 10.

This number can be incrased or decreased depending on your needs but do keep in mind that the more workers you spawn, the more memory you will need.

The queue method has a few default methods built-in but we will be using the 2 basic methods for now. The first method .drain() is called whenever the queue has finished processing. To make debugging easier, we will call that function to alert us whenever the queue is done processing.

q.drain = function () {  
  console.log('The queue has finished processing!');  
};  

Whenever the queue has finished, it will now log a message on the console.

Now for the best part, actually making the queue do something for us.

q.push('http://someurl.com', function (err) {  
  if (err) { return console.log('error in adding tasks to queue'); }  
  console.log('pushed to queue!');  
});

We push the URL we want to process as the first parameter and the callback funtion as the second paramater. When the queue has finished processing, you should 2 messages in the console unless there are more things to process.

finished processing url!
The queue has finished processing!  

You can also pause(), resume() or kill() the queue manually anytime. To find out more about what else the queue can do, check out the documentation on GitHub.