README.md

Why Use Pipeline?

When using pipe from the Node.js streams, errors are not propagated forward through the piped streams, and source streams aren’t closed if a destination stream closed. The pipeline method of the Streams API normalizes these problems, and properly propagates errors of substreams to the pipeline.

A common gulpfile example

A common pattern in gulp files is to simply return a Node.js stream, and expect the gulp tool to handle errors.

// example of a common gulpfile
var gulp = require('gulp');
var uglify = require('gulp-uglify');

gulp.task('compress', function () {
  // returns a Node.js stream, but no handling of error messages
  return gulp.src('lib/*.js')
    .pipe(uglify())
    .pipe(gulp.dest('dist'));
});

There’s an error in one of the JavaScript files, but that error message is the opposite of helpful. You want to know what file and line contains the error. So what is this mess?

When there’s an error in a stream, the Node.js stream fire the 'error' event, but if there’s no handler for this event, it instead goes to the defined uncaught exception handler. The default behavior of the uncaught exception handler is documented:

By default, Node.js handles such exceptions by printing the stack trace to stderr and exiting.

Handling the Errors

Since allowing the errors to make it to the uncaught exception handler isn’t useful, we should handle the exceptions properly. Let’s give that a quick shot.

var gulp = require('gulp');
var uglify = require('gulp-uglify');

gulp.task('compress', function () {
  return gulp.src('lib/*.js')
    .pipe(uglify())
    .pipe(gulp.dest('dist'))
    .on('error', function(err) {
      console.error('Error in compress task', err.toString());
    });
});

Unfortunately, Node.js stream’s pipe function doesn’t forward errors through the chain, so this error handler only handles the errors given by gulp.dest. Instead we need to handle errors for each stream.

var gulp = require('gulp');
var uglify = require('gulp-uglify');

gulp.task('compress', function () {
  function createErrorHandler(name) {
    return function (err) {
      console.error('Error from ' + name + ' in compress task', err.toString());
    };
  }

  return gulp.src('lib/*.js')
    .on('error', createErrorHandler('gulp.src'))
    .pipe(uglify())
    .on('error', createErrorHandler('uglify'))
    .pipe(gulp.dest('dist'))
    .on('error', createErrorHandler('gulp.dest'));
});

This is a lot of complexity to add in each of your gulp tasks, and it’s easy to forget to do it. In addition, it’s still not perfect, as it doesn’t properly signal to gulp’s task system that the task has failed. We can fix this, and we can handle the other pesky issues with error propagations with streams, but it’s even more work!

Using pipelines

The pipeline method is a cheat code of sorts. It’s a wrapper around the pipe functionality that handles these cases for you, so you can stop hacking on your gulpfiles, and get back to hacking new features into your app.

var gulp = require('gulp');
var uglify = require('gulp-uglify');
var pipeline = require('readable-stream').pipeline;

gulp.task('compress', function () {
  return pipeline(
      gulp.src('lib/*.js'),
      uglify(),
      gulp.dest('dist')
  );
});

The pipeline method accepts variable number of streams, which it internally pipes together. It is careful to propagate errors and destroy streams properly. The gulp task system waits for the returned pipeline stream to end, just like before, but can now handle errors from the substreams properly.

Now it’s very clear what plugin the error was from, what the error actually was, and from what file and line number.