Garbage collector doesn't seem to run whenever stdout is piped

[Trinovantes]

I've been playing around to see how fast nodejs' throughput is but I noticed that my program constantly crashes with OOM errors only when I'm piping my stdout to another program rather than a file

const MAX = 0x7FFFFFFFFFFFFFFFn
const BUFFER_SIZE = 32 * 1024

function fizzbuzz() {
    let buffer = ''

    for (let i = 0n; i < MAX; i += 15n) {
        buffer += `${i + 1n}\n${i + 2n}\nFizz\n${i + 4n}\nBuzz\nFizz\n${i + 7n}\n${i + 8n}\nFizz\nBuzz\n${i + 11n}\nFizz\n${i + 13n}\n${i + 14n}\nFizzBuzz\n`

        if (buffer.length > BUFFER_SIZE) {
            process.stdout.write(buffer)
            buffer = ''
        }
    }
}

fizzbuzz()

If I try to run this program with node main.js | pv > /dev/null, the memory usage stays fine for a few seconds before it quickly spikes to the 4 GB heap limit and crashes.

However if I try to run this program with node main.js > /dev/null or node main.js > tmp.txt, it runs just fine and memory usage stays constant.

I've tested on both v16.15.0 and v18.4.0.

[nagayev]

You can move it to issue section.

[Trinovantes]

For whatever reason, piping to another process doesn't release the event loop but piping to a file does

I was able to resolve this by manually releasing the event loop (at a slight performance hit)

async function fizzbuzz() {
    let buffer = ''

    for (let i = 0n; i < MAX; i += 15n) {
        buffer += `${i + 1n}\n${i + 2n}\nFizz\n${i + 4n}\nBuzz\nFizz\n${i + 7n}\n${i + 8n}\nFizz\nBuzz\n${i + 11n}\nFizz\n${i + 13n}\n${i + 14n}\nFizzBuzz\n`

        if (buffer.length > BUFFER_SIZE) {
            process.stdout.write(buffer)
            buffer = ''
            await new Promise((resolve) => setTimeout(resolve, 0))
        }
    }
}

[bnoordhuis]

piping to another process doesn't release the event loop

That's not what happens. You're writing faster than the other end reads, forcing node to buffer.

process.stdout.write() takes a callback as its second argument that's called when the data has been sent. Use that and memory consumption will be stable.

[Trinovantes]

Thanks for the clarification

I'm not sure if pv on the other end of the pipe is the bottleneck but I was able to achieve expected performance with: await new Promise((resolve) => process.stdout.write(buffer, resolve))

[bnoordhuis]

A pipe has a limited capacity, somewhere between 1 and 128 kb. It's basically always possible to write faster than data can flow through it.

Can I suggest you unmark your answer? The phrasing is not entirely correct and using a timeout is not optimal.