DWIM(.nl) - Node Streams


Do What I Mean

January 13th, 2020

Node Streams

I am looking into a way to do ETL (Extract, Transform, Load) in Node. It seams to me that Streams are a very natural way to do this. However, Streams only seem to support simple pipelines of the form:

    const fs = require('fs');
    const zlib = require('zlib');

    fs.createReadStream('./file.txt')                // read a file
      .pipe(zlib.createGzip())                       // zip it
      .pipe(fs.createWriteStream('./file.txt.gz'));  // and write it

In a diagram this would look like:

But imagine I want to zip a file and mark it if it fails an expected checksum. In a diagram, that would look like:

In order to do that, I would need Stream operations like split and join, but these don't seem to exist.

Let's figure out how to do this!

tags: node, research

 Setting up this blog 
 Migrated to org 
Loading script...