Advanced tips for using gulp.js
After getting excited about gulp.js, at some point you need more than the shiny but basic examples. This post discusses some common pitfalls when using gulp.js, plugins and streams in a more advanced and custom way.
In a basic setup, gulp has a nice syntax to use streams and plugins to transform your source files:
This works just fine in many cases, but once you need something more tailored, you may soon face tricky situations. This post addresses some of them.
When using gulp you may have run into the issue of “incompatible streams”. This mostly has to do with the difference of regular streams versus vinyl file objects, and gulp plugins that use libraries supporting only buffers (and not streams).
For example, you can’t pipe a regular Node stream directly to gulp and/or gulp
plugins. Let’s take a read stream, transform the contents using gulp-uglify
and gulp-rename, and finally write the result to disk with
Consider this (erroneous) example:
Why can’t we pipe a read stream to a gulp plugin? Gulp is the streaming build system after all, right? Yes, but the example above ignores the fact that gulp plugins expect Vinyl file objects. You can’t just pipe a read stream to a function (plugin) that expects vinyl file object(s).
The vinyl file object
Gulp uses vinyl-fs, from which it inherits the
gulp.dest() methods. Vinyl-fs uses the vinyl file object, its “virtual
file format”. If we want to use gulp and/or gulp plugins with a regular read
stream, we need to convert the read stream to vinyl first.
A great option is to use vinyl-source-stream, which does exactly that:
The next example starts with a Browserified bundle and eventually converts this to a vinyl stream.
Great. Note that we don’t need to use gulp-rename anymore, since vinyl-source-stream creates a vinyl file instance with the specified filename (which gulp.dest will use to write the bundle).
This gulp method creates a write stream, and is really convenient. It reuses the file names from the read stream, and creates directories (using mkdirp) as necessary. After writing, you can continue piping the stream (e.g. to also gzip the data and write the result to other files).
Streams and buffers
Since you’re interested in using gulp, this post simply assumes you have some
basic knowledge of streams. Vinyl works with virtual files containing either a
buffer or a stream (or
null). With a regular read stream you can listen to
emitted chunks of data:
buffered vinyl file objects back to the
stream. This means you won’t get chunks, but (virtual) files with buffered
contents. The vinyl file format has a
contents property representing a buffer
or a stream, and gulp is using buffers by default:
This clearly shows the data is buffered before the file gets emitted to the stream as a whole.
Gulp uses buffers by default
This is why gulp is using buffered streams by default, since they’re just easier to work with.
In any case, you can tell gulp to pass on a stream for
contents if you set the
buffer option to
false. Here’s a contrived example:
From streams to buffers
Depending on the desired input (and output) stream, and depending on the gulp plugin, you may need to switch from streams to buffers (or vice versa). As said, most plugins work with buffers (although some of them also support streams). Examples include gulp-uglify and gulp-traceur. You can do the conversion to buffers using gulp-buffer:
Or, another contrived example:
From buffers to streams
You can also “streamify” the output of a plugin working with buffers (back) to a read stream by using gulp-streamify or gulp-stream. Then plugins that work (only) with streams can be used before and after the buffer-based plugin:
You don’t need a plugin for everything
Although there are many plugins out there that are very useful and convenient, some tasks and transformations can easily be done without Yet Another Plugin™. Plugins do cause some overhead in that they make you depending on an extra npm module, a plugin interface, (unresponsive?) maintainer, etc. If it’s very easy to do the task at hand without a plugin, or to directly use the original module, then in most cases I would recommend to do so. It’s important to understand the concepts I’ve described above to make the right decision in your situation. Let’s take a look at some examples.
In our examples above we’ve already seen an example of using Browserify directly instead of the (blacklisted) gulp-browserify plugin. The key here is to use vinyl-source-stream (or similar) to allow for regular read streams as input to Vinyl plugins.
Another example is string-based transformations. Here is a very basic plugin to use directly with vinyl buffers:
You could use this plugin like this:
The plugin is unfinished and doesn’t even deal with streams. However, it shows it’s possibly easy to create new transformations using some basic functions. The through2 library is a great wrapper to Node streams and enables transform functions as shown above.
In case you need some custom or dynamic tasks to run, it’s useful to know that gulp is using the Orchestrator module. The gulp.add method is Orchestrator.add (actually all methods are inherited from the Orchestrator module). But, why would you need this?
- You don’t want to clutter the list of gulp tasks with “private” tasks (i.e. not exposing them to the CLI tool).
- You need more dynamic and/or reusable sub-tasks.
Read up on Task automation with npm run to learn more. Just make sure you define clearly what you need on a scale of “build customization”, and what would be the best tool(s) for the job.
However, I think gulp is a great build system that I love to use and really introduced to me the power of streams in Node.js.
Hope this helps! If you have any feedback or additional tips, please let me know in the comments or Twitter: @webprolific.