append to brotli compressed file #207

New Issue

claunia · 2026-01-29T20:39:50Z

claunia commented

2026-01-29 20:39:50 +00:00

Originally created by @worenga on GitHub (Dec 6, 2017).

Is it possible to append data to a brotli compressed file such that it is compressed automatically?

The following would work, e.g. for gzip

$echo 'foo' | gzip >! test.gz
$echo 'foo' | gzip >> test.gz
zcat test
foo
foo

While the same approach would fail using brotli with

corrupt input

Originally created by @worenga on GitHub (Dec 6, 2017). Is it possible to append data to a brotli compressed file such that it is compressed automatically? The following would work, e.g. for gzip ``` $echo 'foo' | gzip >! test.gz $echo 'foo' | gzip >> test.gz zcat test foo foo ``` While the same approach would fail using `brotli` with `corrupt input`

claunia added the feature release-v1.1.1 labels 2026-01-29 20:39:50 +00:00

claunia commented

2026-01-29 20:39:51 +00:00

@eustas commented on GitHub (Dec 6, 2017):

No. Brotli is a bare data stream format, so we decided to make it an error, if something goes after a stream.

The good news, is that we are working on framing format, that (likely) will provide such ability... and much more =) (with some overhead being paid, of course).

@eustas commented on GitHub (Dec 6, 2017): No. Brotli is a bare data stream format, so we decided to make it an error, if something goes after a stream. The good news, is that we are working on framing format, that (likely) will provide such ability... and much more =) (with some overhead being paid, of course).

claunia commented

2026-01-29 20:39:51 +00:00

@danielrh commented on GitHub (Jun 28, 2018):

It might be possible to structure a Brotli stream to have such a property with a few small tweaks to the compressor:

First you'd make sure the last METABLOCK was empty, so it was easy to identify and drop the last Metablock (the final 2 one bits, followed by zeros)
Then you could have an ISUNCOMPRESSED metablock of size two, with the first two bytes of the data stream... so that your priors were correctly chosen
Then set the compressor to ignore matches in the dictionary

With these restrictions you should be able to concatenate them together after dropping the last 2 nonzero bits, right?

You may need a bit more magic, like by inserting a MNIBBLES=0 block to byte-align the metablocks at the end if you don't want to concatenate on the bit-level

@danielrh commented on GitHub (Jun 28, 2018): It might be possible to structure a Brotli stream to have such a property with a few small tweaks to the compressor: - First you'd make sure the last METABLOCK was empty, so it was easy to identify and drop the last Metablock (the final 2 one bits, followed by zeros) - Then you could have an ISUNCOMPRESSED metablock of size two, with the first two bytes of the data stream... so that your priors were correctly chosen - Then set the compressor to ignore matches in the dictionary With these restrictions you should be able to concatenate them together after dropping the last 2 nonzero bits, right? You may need a bit more magic, like by inserting a MNIBBLES=0 block to byte-align the metablocks at the end if you don't want to concatenate on the bit-level

claunia commented

2026-01-29 20:39:51 +00:00

@danielrh commented on GitHub (Oct 30, 2018):

I solved this issue in the drop-in brotli library here https://github.com/dropbox/rust-brotli by doing the above 3 ideas and by disabling the recent items in the distance map.
Programatic usage can be seen here: https://github.com/dropbox/rust-brotli/blob/master/c/catbrotli.c
And the brotli.c file included in this package was modified slightly to pass in the CATABLE option https://github.com/dropbox/rust-brotli/blob/master/c/brotli.c#L412

Hope that helps!

@danielrh commented on GitHub (Oct 30, 2018): I solved this issue in the drop-in brotli library here https://github.com/dropbox/rust-brotli by doing the above 3 ideas and by disabling the recent items in the distance map. Programatic usage can be seen here: https://github.com/dropbox/rust-brotli/blob/master/c/catbrotli.c And the brotli.c file included in this package was modified slightly to pass in the CATABLE option https://github.com/dropbox/rust-brotli/blob/master/c/brotli.c#L412 Hope that helps!

claunia commented

2026-01-29 20:39:51 +00:00

@eustas commented on GitHub (Oct 30, 2018):

The problem with "catable" brotli is that it is impossible to prove that given file is "catable" without fully decompressing it.
Internally we have encoder that allows parallel encoding, i.e. encoders could produce streams that could be appended one after another... And going to publish that soon. Of course that is different story... But it does not ignore matches in dictionary and this might produce denser stream.

BTW, thank you, Daniel for developing rust-brotli, that is awesome project!

@eustas commented on GitHub (Oct 30, 2018): The problem with "catable" brotli is that it is impossible to prove that given file is "catable" without fully decompressing it. Internally we have encoder that allows parallel encoding, i.e. encoders could produce streams that could be appended one after another... And going to publish that soon. Of course that is different story... But it does not ignore matches in dictionary and this might produce denser stream. BTW, thank you, Daniel for developing rust-brotli, that is awesome project!

claunia commented

2026-01-29 20:39:52 +00:00

@danielrh commented on GitHub (Oct 31, 2018):

Yes it's true that it is impossible to verify without decompressing. That is one of the main reasons I added this header magic number as the first metadata metablock: the header contains information about whether the file was designed to concatenate. Of course that's advisory, you would still need to decompress to fully verify. It's still likely faster than compressing the concatenated chunk. Perhaps the default mode should refuse to concatenate the file if the header is missing. It has some heuristics already to look for 'concatability', which rule out files generated with default brotli-like tools.

And you are correct: rust-brotli uses this catable flag internally to make multithreaded files. I think for internally created files, it could allow dictionary usage from any of the threads since we prepend the previous parts of the file to the ring buffer, so naturally it should look farther for the dictionary, but I haven't tried that mode of operation

It doesn't seem to me that splitting a file N ways often results in Nx improvement: it appears that certain parts of the file require significantly more CPU time than other parts. I haven't profiled the compression much yet.

@danielrh commented on GitHub (Oct 31, 2018): Yes it's true that it is impossible to verify without decompressing. That is one of the main reasons I added this header magic number as the first metadata metablock: the header contains information about whether the file was designed to concatenate. Of course that's advisory, you would still need to decompress to fully verify. It's still likely faster than compressing the concatenated chunk. Perhaps the default mode should refuse to concatenate the file if the header is missing. It has some heuristics already to look for 'concatability', which rule out files generated with default brotli-like tools. And you are correct: rust-brotli uses this catable flag internally to make multithreaded files. I think for internally created files, it could allow dictionary usage from any of the threads since we prepend the previous parts of the file to the ring buffer, so naturally it should look farther for the dictionary, but I haven't tried that mode of operation It doesn't seem to me that splitting a file N ways often results in Nx improvement: it appears that certain parts of the file require significantly more CPU time than other parts. I haven't profiled the compression much yet.

claunia commented

2026-01-29 20:39:52 +00:00

@rifler commented on GitHub (Jan 13, 2021):

The good news, is that we are working on framing format, that (likely) will provide such ability... and much more =) (with some overhead being paid, of course).

Hi, can you please provide issues/plans, where we can read about this?

@rifler commented on GitHub (Jan 13, 2021): > The good news, is that we are working on framing format, that (likely) will provide such ability... and much more =) (with some overhead being paid, of course). Hi, can you please provide issues/plans, where we can read about this?

claunia commented

2026-01-29 20:39:53 +00:00

@s-sols commented on GitHub (Mar 23, 2021):

No. Brotli is a bare data stream format, so we decided to make it an error, if something goes after a stream.

The good news, is that we are working on framing format, that (likely) will provide such ability... and much more =) (with some overhead being paid, of course).

There is a need to combine several precompressed chunks. This chunks might be precompressed in the special format that allows concatenating in any sequence. This need relates to output precached content from webserver.

Have any related features been implemented in the library?

Thanks.

@s-sols commented on GitHub (Mar 23, 2021): > No. Brotli is a bare data stream format, so we decided to make it an error, if something goes after a stream. > > The good news, is that we are working on framing format, that (likely) will provide such ability... and much more =) (with some overhead being paid, of course). There is a need to combine several precompressed chunks. This chunks might be precompressed in the special format that allows concatenating in any sequence. This need relates to output precached content from webserver. Have any related features been implemented in the library? Thanks.

claunia commented

2026-01-29 20:39:54 +00:00

@danielrh commented on GitHub (Mar 23, 2021):

@s-sols : I have created a binary-compatible brotli library here with a new option flag to create concattable files here:
https://github.com/dropbox/rust-brotli by doing the above 3 ideas and by disabling the recent items in the distance map.
Programatic usage can be seen here: https://github.com/dropbox/rust-brotli/blob/master/c/catbrotli.c
And the brotli.c file included in this package was modified slightly to pass in the CATABLE option https://github.com/dropbox/rust-brotli/blob/master/c/brotli.c#L412
If you don't have access to a rust compiler, c code can be created through use of the mrustc package. lmk if you have issues

@danielrh commented on GitHub (Mar 23, 2021): @s-sols : I have created a binary-compatible brotli library here with a new option flag to create concattable files here: https://github.com/dropbox/rust-brotli by doing the above 3 ideas and by disabling the recent items in the distance map. Programatic usage can be seen here: https://github.com/dropbox/rust-brotli/blob/master/c/catbrotli.c And the brotli.c file included in this package was modified slightly to pass in the CATABLE option https://github.com/dropbox/rust-brotli/blob/master/c/brotli.c#L412 If you don't have access to a rust compiler, c code can be created through use of the mrustc package. lmk if you have issues

claunia commented

2026-01-29 20:39:54 +00:00

@dgtlmoon commented on GitHub (Nov 2, 2022):

@danielrh hey super cool work! I got the c libraries to compile without problem

question about python and appending to an existing brotli "stream" https://github.com/dropbox/rust-brotli/blob/master/c/py/brotli_test.py

how would that work?

Are you able to add a that py test case there in that src?

Not quite sure where that existing compressed brotli would get passed to BrotliCompress

        output = BrotliCompress(self.test_data,
                                {
                                    BROTLI_PARAM_QUALITY:5,
                                    BROTLI_PARAM_CATABLE:1,
                                    BROTLI_PARAM_MAGIC_NUMBER:1,
                                },
                                1)
        rt = BrotliDecode(output, 2)

btw https://lib.rs/crates/brotli-ffi

@dgtlmoon commented on GitHub (Nov 2, 2022): @danielrh hey super cool work! I got the c libraries to compile without problem question about python and appending to an existing brotli "stream" https://github.com/dropbox/rust-brotli/blob/master/c/py/brotli_test.py how would that work? Are you able to add a that py test case there in that src? Not quite sure where that existing compressed brotli would get passed to `BrotliCompress` ```python output = BrotliCompress(self.test_data, { BROTLI_PARAM_QUALITY:5, BROTLI_PARAM_CATABLE:1, BROTLI_PARAM_MAGIC_NUMBER:1, }, 1) rt = BrotliDecode(output, 2) ``` btw https://lib.rs/crates/brotli-ffi

claunia commented

2026-01-29 20:39:55 +00:00

@eustas commented on GitHub (Jun 20, 2023):

Hmm. Reconsidering. While brotli stream does not like "tails" we can overcome that in CLI. Will see if we can have it in v1.1

@eustas commented on GitHub (Jun 20, 2023): Hmm. Reconsidering. While brotli stream does not like "tails" we can overcome that in CLI. Will see if we can have it in v1.1

claunia referenced this issue

2026-01-29 20:50:59 +00:00

[PR #207] [MERGED] [types.h] make std integer types for _MSC_VER compatible with CFFI #803

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#207