Brotli framing format requirements #143

New Issue

claunia · 2026-01-29T20:38:23Z

claunia commented

2026-01-29 20:38:23 +00:00

Originally created by @eustas on GitHub (Oct 27, 2016).

Please write your thoughts on what features Brotli framing format should have.
PS: framing format should not be a "swiss-knife", i.e. match every imaginable use case.

Originally created by @eustas on GitHub (Oct 27, 2016). Please write your thoughts on what features Brotli framing format should have. PS: framing format should not be a "swiss-knife", i.e. match every imaginable use case.

claunia closed this issue

2026-01-29 20:38:24 +00:00

claunia commented

2026-01-29 20:38:25 +00:00

@eustas commented on GitHub (Oct 27, 2016):

Easy pick:

should not be a valid UTF-8 string
should not be a valid brotli stream
should have unique "magic bytes" signature

First 2 requirements give us that the first "magic byte" is 0x91.

@eustas commented on GitHub (Oct 27, 2016): Easy pick: - should not be a valid UTF-8 string - should not be a valid brotli stream - should have unique "magic bytes" signature First 2 requirements give us that the first "magic byte" is `0x91`.

claunia commented

2026-01-29 20:38:26 +00:00

@dsnet commented on GitHub (Oct 27, 2016):

I came up with a possible design here: brotli-framing-format.md

High level ideas:

Designed to be more efficient with random-access decompression by coalescing records together into a unified index (like xz), rather than having the offsets be distributed throughout the file (like lzip).
I don't feel strongly that the format should provide ways to encode the original filename, owner, group, modified date, etc. My opinion is that an archive format like tar or zip should be used and that the framed format is only responsible for compression.
No option to choose between multiple checksum methods (just use CRC-32). If a user cares about data integrity, they can encrypt and sign the file afterwards.
Limited support for parallel decompression when reading the input forwards as a stream. The "sync markers" to do this also provide limited forms of data recovery with partial files.
The StaticDict feature is experimental. I'm not sure if this should be part of the format, but it seems like an interesting idea to me.

You may want to consider giving the framed format a completely different name than simply "brotli framing format". There may be much confusion between what the difference is between two.

@dsnet commented on GitHub (Oct 27, 2016): I came up with a possible design here: [brotli-framing-format.md](https://github.com/dsnet/compress/blob/master/doc/brotli-framing-format.md) High level ideas: - Designed to be more efficient with random-access decompression by coalescing records together into a unified index (like [xz](http://tukaani.org/xz/format.html)), rather than having the offsets be distributed throughout the file (like [lzip](http://lzip.nongnu.org/manual/lzip_manual.html#File-format)). - I don't feel strongly that the format should provide ways to encode the original filename, owner, group, modified date, etc. My opinion is that an archive format like tar or zip should be used and that the framed format is only responsible for compression. - No option to choose between multiple checksum methods (just use CRC-32). If a user cares about data integrity, they can encrypt and sign the file afterwards. - Limited support for parallel decompression when reading the input forwards as a stream. The "sync markers" to do this also provide limited forms of data recovery with partial files. - The StaticDict feature is experimental. I'm not sure if this should be part of the format, but it seems like an interesting idea to me. You may want to consider giving the framed format a completely different name than simply "brotli framing format". There may be much confusion between what the difference is between two.

claunia commented

2026-01-29 20:38:27 +00:00

@dsnet commented on GitHub (Oct 27, 2016):

should not be a valid UTF-8 string

should not be a valid brotli stream

With 0x91 as the first byte, if the second byte doesn't have the highest bit set, we can guarantee that it isn't UTF-8 either.

@dsnet commented on GitHub (Oct 27, 2016): > - should not be a valid UTF-8 string > - should not be a valid brotli stream With `0x91` as the first byte, if the second byte doesn't have the highest bit set, we can guarantee that it isn't UTF-8 either.

claunia commented

2026-01-29 20:38:28 +00:00

@buu700 commented on GitHub (May 21, 2017):

@eustas asked me to move my feature request from #549 to this thread:

Would it be practical for a future version of Brotli to detect strings of formats like base64 and hex and decode them to binary before compressing / re-encode them as part of the decompression process?

I noticed just now during a comparison between compressing a binary file and the equivalent base64 (https://github.com/kripken/emscripten/issues/5104#issuecomment-302939570) that the base64 version ended up larger, which it occurs to me shouldn't need to be the case given that the two contain almost exactly the same information.

Since I don't know anything about Brotli's internals, I have nothing more specific to suggest in terms of the framing format design.

@buu700 commented on GitHub (May 21, 2017): @eustas asked me to move my feature request from #549 to this thread: >Would it be practical for a future version of Brotli to detect strings of formats like base64 and hex and decode them to binary before compressing / re-encode them as part of the decompression process? > >I noticed just now during a comparison between compressing a binary file and the equivalent base64 (https://github.com/kripken/emscripten/issues/5104#issuecomment-302939570) that the base64 version ended up larger, which it occurs to me shouldn't need to be the case given that the two contain almost exactly the same information. Since I don't know anything about Brotli's internals, I have nothing more specific to suggest in terms of the framing format design.

claunia commented

2026-01-29 20:38:30 +00:00

@stokito commented on GitHub (Nov 10, 2018):

As I understood the format is already developed and used internally, right?
Can you confirm that this is a current spec of the format https://github.com/madler/brotli/blob/master/br-format-v3.txt

@stokito commented on GitHub (Nov 10, 2018): As I understood the format is already developed and used internally, right? Can you confirm that this is a current spec of the format https://github.com/madler/brotli/blob/master/br-format-v3.txt

claunia commented

2026-01-29 20:38:31 +00:00

@jason-sachs commented on GitHub (Nov 4, 2020):

Since this issue was closed, where is the framing format documented?

@jason-sachs commented on GitHub (Nov 4, 2020): Since this issue was closed, where is the framing format documented?

claunia commented

2026-01-29 20:38:31 +00:00

@jason-sachs commented on GitHub (Nov 4, 2020):

I have used brotli.exe in the Python distribution brotli 1.0.9 and it doesn't appear to include framing or to have any option to include framing. (for example, a zero-byte input compresses to hex A1 01)

@jason-sachs commented on GitHub (Nov 4, 2020): I have used brotli.exe in the Python distribution brotli 1.0.9 and it doesn't appear to include framing or to have any option to include framing. (for example, a zero-byte input compresses to hex `A1 01`)

claunia referenced this issue

2026-01-29 20:50:43 +00:00

[PR #143] [CLOSED] Fix all Microsoft Visual Studio bugs and errors and Add Microsoft Visual Studio Projectfiles #755

claunia referenced this issue

2026-01-29 20:50:44 +00:00

[PR #143] Fix all Microsoft Visual Studio bugs and errors and Add Microsoft Visual Studio Projectfiles #758

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#143