Do What I Mean
January 13th, 2020
I am looking into a way to do ETL (Extract, Transform, Load) in Node. It seams to me that Streams are a very natural way to do this. However, Streams only seem to support simple pipelines of the form:
const fs = require('fs');
const zlib = require('zlib');
fs.createReadStream('./file.txt') // read a file
.pipe(zlib.createGzip()) // zip it
.pipe(fs.createWriteStream('./file.txt.gz')); // and write it
In a diagram this would look like:
But imagine I want to zip a file and mark it if it fails an expected checksum. In a diagram, that would look like:
In order to do that, I would need Stream operations like split
and join
, but these don't seem to exist.
Let's figure out how to do this!