Add library and cli flags for file format with embedded dictionary #508

New Issue

claunia · 2026-01-29T20:44:58Z

claunia commented

2026-01-29 20:44:58 +00:00

Originally created by @pmeenan on GitHub (Apr 29, 2024).

This is still in flight but I wanted to get some feedback from the tooling side before we go too far on the IETF spec for dictionary-compressed responses.

We are considering creating a new file/stream format that adds a 35-byte header before the compressed stream with a magic signature (DCB) and sha-256 hash of the dictionary that was used to compress the resource.

Currently the dictionary hash is sent in a separate header but there may be value in putting the hash in the file itself and removing the need for an extra header.

Optimally, if we go down this path it would be useful for the brotli cli and API's to support generating and decompressing these streams directly rather than wrapping their output in more tooling.

On compression:

Add a flag for generating "Dictionary-Compressed Brotli"
Flag limits the compression window to 16 MB
Flag requires a dictionary to be specified
Output stream is prefixed with DCB + sha-256 hash of dictionary

On decompression:

Add a flag for decompressing "Dictionary-Compressed Brotli" (or autodetect from the magic signature)
Flag requires a dictionary be specified
Flag sets compression window max to 16 MB
On decompression, if the hash doesn't match the provided dictionary, fail

Does this sound reasonable and make sense to add if we do go down the route of specifying a stream prefix for the dictionary-compressed streams? Are there any concerns/suggestions on the plan itself?

Originally created by @pmeenan on GitHub (Apr 29, 2024). This is still in flight but I wanted to get some feedback from the tooling side before we go too far on the IETF spec for dictionary-compressed responses. We [are considering](https://github.com/httpwg/http-extensions/issues/2770) creating a new file/stream format that adds a 35-byte header before the compressed stream with a magic signature (`DCB`) and sha-256 hash of the dictionary that was used to compress the resource. Currently the dictionary hash is sent in a separate header but there may be value in putting the hash in the file itself and removing the need for an extra header. Optimally, if we go down this path it would be useful for the brotli cli and API's to support generating and decompressing these streams directly rather than wrapping their output in more tooling. On compression: * Add a flag for generating "Dictionary-Compressed Brotli" * Flag limits the compression window to 16 MB * Flag requires a dictionary to be specified * Output stream is prefixed with `DCB` + sha-256 hash of dictionary On decompression: * Add a flag for decompressing "Dictionary-Compressed Brotli" (or autodetect from the magic signature) * Flag requires a dictionary be specified * Flag sets compression window max to 16 MB * On decompression, if the hash doesn't match the provided dictionary, fail Does this sound reasonable and make sense to add if we do go down the route of specifying a stream prefix for the dictionary-compressed streams? Are there any concerns/suggestions on the plan itself?

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#508