Set your Publisher ID in src/config/ads.ts to display live ads.
Streams, Buffers & Handling Large Data
Processing multi-gigabyte data sets with minimal memory footprint
When working with large files (videos, logs, datasets), loading the entire payload into RAM will exhaust system memory. Node.js Streams allow you to process data chunk-by-chunk in real time.
The Four Stream Types and Buffers
Buffer: Represents fixed-size, raw binary data allocated outside the V8 heap memory.
Readable Stream: Source of data you can read from (e.g., fs.createReadStream, incoming HTTP requests).
Writable Stream: Destination for data (e.g., fs.createWriteStream, HTTP responses).
Transform Stream: Can modify data as it is written and read (e.g., zlib.createGzip()).
The .pipe() method connects a readable stream directly to a writable destination, automatically managing backpressure.
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';
async function compressLogFile(sourcePath, destPath) {
try {
const source = createReadStream(sourcePath);
const destination = createWriteStream(destPath);
const gzip = createGzip();
// pipeline handles automatic cleanup and error forwarding
await pipeline(source, gzip, destination);
console.log('File compressed successfully without excessive RAM usage!');
} catch (err) {
console.error('Pipeline failed:', err);
}
}
compressLogFile('./access.log', './access.log.gz');Set your Publisher ID in src/config/ads.ts to display live ads.
Test Yourself With Exercise
Which helper function from node:stream/promises connects streams while handling error cleanup and backpressure automatically?