BrotliDecompressedSize fails for certain output sizes #118

New Issue

claunia · 2026-01-29T20:35:45Z

claunia commented

2026-01-29 20:35:45 +00:00

Originally created by @jeroen on GitHub (Jun 5, 2016).

A user of the brotli R library has reported strange behavior of BrotliDecompressedSize failing for certain combinations of data and compression size.

I put a few example files here. I compressed the file uncompressed with quality 3, 4, and 5 and default settings otherwise mode = "generic", log_win = 22, log_block = 0.

git clone https://gist.github.com/jeroenooms/9fe37c9f04ad7f054dc2901a0bc5edaa

If I try to decompress the file compressed3 via the C interface, the call to BrotliDecompressedSize fails. It does not happen for compressed4 and compressed5. Also it does not seem to happen for the bro command line utility.

Originally created by @jeroen on GitHub (Jun 5, 2016). A user of the brotli R library has [reported](https://github.com/jeroenooms/brotli/issues/1) strange behavior of `BrotliDecompressedSize` failing for certain combinations of data and compression size. I put a few example files [here](https://gist.github.com/jeroenooms/9fe37c9f04ad7f054dc2901a0bc5edaa). I compressed the file `uncompressed` with quality 3, 4, and 5 and default settings otherwise `mode = "generic"`, `log_win = 22`, `log_block = 0`. ``` sh git clone https://gist.github.com/jeroenooms/9fe37c9f04ad7f054dc2901a0bc5edaa ``` If I try to decompress the file `compressed3` via the C interface, the call to `BrotliDecompressedSize` fails. It does not happen for `compressed4` and `compressed5`. Also it does not seem to happen for the `bro` command line utility.

claunia closed this issue

2026-01-29 20:35:46 +00:00

claunia commented

2026-01-29 20:35:47 +00:00

@eustas commented on GitHub (Jun 7, 2016):

There is a very limited number of cases in which BrotliDecompressedSize works.

The whole input is compressed in a single block
The whole input is a single uncompressed block

The length of both compressed and uncompressed block is limited to 1 << 24 == 16MiB.

Technically it is possible to make this function work for a case: multiple uncompressed blocks followed by 0 or 1 compressed block.

@eustas commented on GitHub (Jun 7, 2016): There is a very limited number of cases in which BrotliDecompressedSize works. 1) The whole input is compressed in a single block 2) The whole input is a single uncompressed block The length of both compressed and uncompressed block is limited to 1 << 24 == 16MiB. Technically it is possible to make this function work for a case: multiple uncompressed blocks followed by 0 or 1 compressed block.

claunia commented

2026-01-29 20:35:49 +00:00

@eustas commented on GitHub (Jun 7, 2016):

Different compression levels affect ability of encoder to detect better input splitting...

@eustas commented on GitHub (Jun 7, 2016): Different compression levels affect ability of encoder to detect better input splitting...

claunia commented

2026-01-29 20:35:51 +00:00

@jeroen commented on GitHub (Jun 7, 2016):

So how is the client supposed to know how much memory to allocate for BrotliDecompressBuffer if we cannot rely on BrotliDecompressedSize ?

@jeroen commented on GitHub (Jun 7, 2016): So how is the client supposed to know how much memory to allocate for `BrotliDecompressBuffer` if we cannot rely on `BrotliDecompressedSize` ?

claunia commented

2026-01-29 20:35:52 +00:00

@eustas commented on GitHub (Jun 7, 2016):

Brotli is a low level format. It is designed for streaming processing.
We are planning to add framing format that will allow things like stream navigation, reliable calculation of uncompressed size, consistency checks, etc.
Unfortunately, it is not ready yet...

@eustas commented on GitHub (Jun 7, 2016): Brotli is a low level format. It is designed for streaming processing. We are planning to add framing format that will allow things like stream navigation, reliable calculation of uncompressed size, consistency checks, etc. Unfortunately, it is not ready yet...

claunia commented

2026-01-29 20:35:53 +00:00

@jeroen commented on GitHub (Jun 22, 2016):

Perhaps this is mostly a documentation bug then. Without fully understanding technical details of brotli, it seemed the basic API decompressing a buffer was to allocate memory based on BrotliDecompressedSize and then invoke BrotliDecompressBuffer.

I maintain brotli bindings for the R language. So I suppose I better switch to the BrotliDecompressStream API to decompress arbitrary brotli data?

Can I still use BrotliCompressBuffer for encoding a buffer or does this API have similar limitations?

@jeroen commented on GitHub (Jun 22, 2016): Perhaps this is mostly a documentation bug then. Without fully understanding technical details of brotli, it seemed the basic API decompressing a buffer was to allocate memory based on `BrotliDecompressedSize` and then invoke `BrotliDecompressBuffer`. I maintain [brotli bindings for the R language](https://github.com/jeroenooms/brotli/blob/master/src/wrapper.cc). So I suppose I better switch to the `BrotliDecompressStream` API to decompress arbitrary brotli data? Can I still use `BrotliCompressBuffer` for encoding a buffer or does this API have similar limitations?

claunia commented

2026-01-29 20:35:54 +00:00

@eustas commented on GitHub (Jun 24, 2016):

Hello.

Just recently updated BrotliDecompressedSize. Now it supports multiple non-compressed and metadata blocks. Unfortunately, until brotli framing format is implemented, it will never cover all the cases. So, currently, I would recommend to use BrotliDecompressStream.

As for encoder, BrotliEncoderMaxCompressedSize provides the exact answer about how large output may be; but it only works for BrotliEncoderCompress.

Thank you for the report and for the R bindings.
Best regards,
Eugene.

@eustas commented on GitHub (Jun 24, 2016): Hello. Just recently updated `BrotliDecompressedSize`. Now it supports multiple non-compressed and metadata blocks. Unfortunately, until brotli framing format is implemented, it will never cover all the cases. So, currently, I would recommend to use `BrotliDecompressStream`. As for encoder, `BrotliEncoderMaxCompressedSize` provides the exact answer about how large output may be; but it only works for `BrotliEncoderCompress`. Thank you for the report and for the R bindings. Best regards, Eugene.

claunia commented

2026-01-29 20:35:55 +00:00

@jeroen commented on GitHub (Jun 27, 2016):

Have you pushed the updates for BrotliDecompressedSize yet?

@jeroen commented on GitHub (Jun 27, 2016): Have you pushed the updates for `BrotliDecompressedSize` yet?

claunia commented

2026-01-29 20:35:56 +00:00

@eustas commented on GitHub (Jun 27, 2016):

Yup, in PR #386

@eustas commented on GitHub (Jun 27, 2016): Yup, in PR #386

claunia referenced this issue

2026-01-29 20:50:36 +00:00