0.6.0 compression is slow (or never finishes?) for data over ~1.57MB #189

New Issue

claunia · 2026-01-29T20:39:32Z

claunia commented

2026-01-29 20:39:32 +00:00

Originally created by @jotto on GitHub (Aug 11, 2017).

It might have to do with randomness in the file - I can easily compress a large file of zeros, but not JPGs over ~2MB or /dev/random over ~1572864 bytes.

// brotli.js
const fs = require("fs");
const compressSync = require("iltorb").compressSync;

const args = process.argv.slice(2);
const filename = args[0];

console.time("compress");
const output = compressSync(fs.readFileSync(filename));
console.timeEnd("compress");
console.log(output.length, "bytes");

// increase to 1572865 to see it fail/stall (or you might need to go a little higher)
dd if=/dev/urandom bs=1 count=1572864 > brotli.txt
node brotli.js brotli.txt

Originally created by @jotto on GitHub (Aug 11, 2017). It might have to do with randomness in the file - I can easily compress a large file of zeros, but not JPGs over ~2MB or /dev/random over ~1572864 bytes. ```javascript // brotli.js const fs = require("fs"); const compressSync = require("iltorb").compressSync; const args = process.argv.slice(2); const filename = args[0]; console.time("compress"); const output = compressSync(fs.readFileSync(filename)); console.timeEnd("compress"); console.log(output.length, "bytes"); ``` ```bash // increase to 1572865 to see it fail/stall (or you might need to go a little higher) dd if=/dev/urandom bs=1 count=1572864 > brotli.txt node brotli.js brotli.txt ```

claunia closed this issue

2026-01-29 20:39:32 +00:00

claunia commented

2026-01-29 20:39:33 +00:00

@eustas commented on GitHub (Aug 11, 2017):

Going to investigate it on Monday.

@eustas commented on GitHub (Aug 11, 2017): Going to investigate it on Monday.

claunia commented

2026-01-29 20:39:33 +00:00

@eustas commented on GitHub (Aug 23, 2017):

Haven't done investivation, just inspected the code.
This loop looks suspicious: https://github.com/MayhemYDG/iltorb/blob/master/src/enc/stream_encode_worker.cc#L25
See - no output space is given; if it happens, that internal brotli output buffer is full - it can't continue processing... so it becomes infinite loop...
Outer wrapper also doesn't care, if chunk is too big to make "StreamEncodeWorker" choke...

@eustas commented on GitHub (Aug 23, 2017): Haven't done investivation, just inspected the code. This loop looks suspicious: https://github.com/MayhemYDG/iltorb/blob/master/src/enc/stream_encode_worker.cc#L25 See - no output space is given; if it happens, that internal brotli output buffer is full - it can't continue processing... so it becomes infinite loop... Outer wrapper also doesn't care, if chunk is too big to make "StreamEncodeWorker" choke...

claunia referenced this issue

2026-01-29 20:50:55 +00:00

[PR #189] [MERGED] Move literal cost computation to where it's used. #789

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#189