stream and readline: Handling Data in Pieces

Objectives

By the end of this lesson, you should be able to:

• Read a file as a stream, handling data as chunks instead of all at once
• Pipe a readable stream directly into a writable stream
• Read input line by line with readline

💡 Why this matters: Loading an entire large file into memory at once (fs.readFileSync) works fine for small files, but not for a multi-gigabyte log file or video upload. Streams process data in small chunks as it arrives, keeping memory usage low regardless of the total size, this is exactly how Express handles large request bodies and file uploads under the hood.

⚠️ A note on verification: every snippet and output in this lesson was actually run with Node.js.

Reading a File as a Stream

const fs = require('fs');

const line = 'line of sample text for streaming demonstration\n';
fs.writeFileSync('big.txt', line.repeat(20));

const readStream = fs.createReadStream('big.txt', { encoding: 'utf8', highWaterMark: 256 });

let chunkCount = 0;
readStream.on('data', (chunk) => {
  chunkCount++;
  console.log(`received chunk ${chunkCount}, length ${chunk.length}`);
});

readStream.on('end', () => {
  console.log('stream finished, total chunks:', chunkCount);
});

received chunk 1, length 256
received chunk 2, length 256
received chunk 3, length 256
received chunk 4, length 192
stream finished, total chunks: 4

fs.createReadStream reads a file in pieces instead of all at once, each piece triggers a 'data' event with that chunk. The highWaterMark option here (256 bytes) is set artificially small just to make the chunking visible, in real use it defaults to 64KB, large enough that most files still arrive in very few chunks, but small enough that a multi-gigabyte file never needs to fit in memory all at once.

Piping Streams Together

const fs = require('fs');

const readStream = fs.createReadStream('source.txt');
const writeStream = fs.createWriteStream('destination.txt');

readStream.pipe(writeStream);

writeStream.on('finish', () => {
  console.log('copy complete');
  console.log(fs.readFileSync('destination.txt', 'utf8'));
});

copy complete
Some content to copy via streams.

.pipe() connects a readable stream directly to a writable one, data flows from source to destination automatically, chunk by chunk, without ever loading the entire file into memory. The 'finish' event on the write stream fires once every chunk has been written. This is the same mechanism Express uses internally when streaming a file response back to a client.

Reading Input Line by Line with readline

const readline = require('readline');
const fs = require('fs');

fs.writeFileSync('lines.txt', 'apple\nbanana\ncherry\n');

const rl = readline.createInterface({
  input: fs.createReadStream('lines.txt'),
  crlfDelay: Infinity
});

let lineNumber = 0;
rl.on('line', (line) => {
  lineNumber++;
  console.log(`line ${lineNumber}: ${line}`);
});

rl.on('close', () => {
  console.log('done reading, total lines:', lineNumber);
});

line 1: apple
line 2: banana
line 3: cherry
done reading, total lines: 3

readline.createInterface takes a stream as its input (here, a file read stream, it works identically with process.stdin for interactive command-line input), and emits a 'line' event for every complete line, 'close' fires once the input is exhausted. This confirms readline is genuinely built on streams, not a separate mechanism.

readline for Interactive Input

const readline = require('readline');

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout
});

rl.question('What is your name? ', (answer) => {
  console.log(`Hello, ${answer}!`);
  rl.close();
});

Run interactively and typing Priya:

What is your name? Hello, Priya!

.question(prompt, callback) prints a prompt, waits for the user to type a line and press Enter, then calls back with the answer, the standard building block for a simple command-line tool that needs user input.

Try It

1. Create a text file of at least 500 characters, read it with fs.createReadStream using a small highWaterMark (like 100), and count how many chunks it arrives in.
2. Use .pipe() to copy one file to another, and confirm the destination file’s contents exactly match the source afterward.
3. Use readline to read a file line by line, and log only lines longer than 5 characters.
4. Write an interactive script using readline.question() that asks for a number and logs whether it’s even or odd.

Recap

• Streams process data in chunks as it arrives ('data'/'end' events), keeping memory usage low regardless of file size.
• .pipe() connects a readable stream directly to a writable one, the mechanism behind Express’s own file-serving and upload handling.
• readline.createInterface reads line by line from any stream, including process.stdin for interactive command-line input.

Next lesson: this module’s exercises, tying together every core module covered.