What are streams in Node.js, what types exist, and why would you use one instead of reading a whole file?

A stream processes data in chunks as it arrives, instead of loading everything into memory first. Reading a two gigabyte file with fs.readFile needs two gigabytes of RAM; reading it as a stream needs only the size of one chunk.

The four types:

• Readable — a source you consume from, such as fs.createReadStream or an incoming HTTP request.
• Writable — a destination, such as fs.createWriteStream or an HTTP response.
• Duplex — both, with the two sides independent. A TCP socket is the standard example.
• Transform — a duplex stream where the output is a function of the input, such as zlib.createGzip().

Backpressure is the reason streams are worth understanding. If the destination is slower than the source, data piles up in memory. pipe, and better still pipeline, handle this for you by pausing the source when the destination's buffer is full.

const { pipeline } = require('stream/promises');
await pipeline(
  fs.createReadStream('in.csv'),
  zlib.createGzip(),
  fs.createWriteStream('out.csv.gz')
);

Note: Prefer pipeline over pipe. pipe does not forward errors or clean up the remaining streams when one fails, which leaks file descriptors.