Custom dictionaries below level 5 #502

New Issue

claunia · 2026-01-29T20:44:56Z

claunia commented

2026-01-29 20:44:56 +00:00

Originally created by @rachel-bousfield on GitHub (Mar 26, 2024).

Custom dictionaries can be attached during compression and decompression using C APIs like BrotliEncoderAttachPreparedDictionary. However, it appears they aren't used below brotli level 5. This causes a kind of silent failure where the user doesn't observe the lack of improvement.

There's a few solutions to this issue

The attach and/or prepare methods could document the behavior.
These and BrotliEncoderPrepareDictionary could fail when an incompatible level is applied.
The dictionary format could include the minimum brotli level for compatibility & API-simplification reasons.
Decide this is a bug and implement the feature for lower levels (not backwards compatible)

Here's an example in case the issue isn't clear

brotli -0 -D dictionary.lz -o dict
brotli -0 no-dict
diff dict no-dict    # would use <() but this doesn't work either

Originally created by @rachel-bousfield on GitHub (Mar 26, 2024). Custom dictionaries can be attached during compression and decompression using C APIs like `BrotliEncoderAttachPreparedDictionary`. However, [it appears they aren't used below brotli level 5](https://blog.cloudflare.com/brotli-compression-using-a-reduced-dictionary). This causes a kind of silent failure where the user doesn't observe the lack of improvement. There's a few solutions to this issue 1. The attach and/or prepare methods could document the behavior. 2. These and `BrotliEncoderPrepareDictionary` could fail when an incompatible level is applied. 3. The dictionary format could include the minimum brotli level for compatibility & API-simplification reasons. 4. Decide this is a bug and implement the feature for lower levels (not backwards compatible) Here's an example in case the issue isn't clear ```sh brotli -0 -D dictionary.lz -o dict brotli -0 no-dict diff dict no-dict # would use <() but this doesn't work either ```

claunia commented

2026-01-29 20:44:56 +00:00

@eustas commented on GitHub (Mar 26, 2024):

Thanks for reporting. I'll check and take action when I get spare cycles.
Agree, that at least CLI should let users know if dictionary is ignored.

@eustas commented on GitHub (Mar 26, 2024): Thanks for reporting. I'll check and take action when I get spare cycles. Agree, that at least CLI should let users know if dictionary is ignored.

claunia commented

2026-01-29 20:44:57 +00:00

@rachel-bousfield commented on GitHub (Mar 26, 2024):

I'd also look into how the C API could be improved. That's actually how I first discovered this issue. I was rather confused why my dictionary wasn't working :)

@rachel-bousfield commented on GitHub (Mar 26, 2024): I'd also look into how the C API could be improved. That's actually how I first discovered this issue. I was rather confused why my dictionary wasn't working :)

claunia commented

2026-01-29 20:44:57 +00:00

@pmeenan commented on GitHub (Apr 9, 2024):

FWIW, ZStandard allows for dictionary use down to 0 (though, practically you don't see big benefit until 2-3).

I don't know enough about brotli's actual encoding to know if it makes sense, but it could be useful to allow for dictionaries to work at the lower levels as well if that allows for lower CPU use but still decent savings.

@pmeenan commented on GitHub (Apr 9, 2024): FWIW, ZStandard allows for dictionary use down to 0 (though, practically you don't see big benefit until 2-3). I don't know enough about brotli's actual encoding to know if it makes sense, but it could be useful to allow for dictionaries to work at the lower levels as well if that allows for lower CPU use but still decent savings.

claunia referenced this issue

2026-01-29 20:52:41 +00:00

[PR #505] [MERGED] Fix #502 decoder bug #1145

claunia referenced this issue

2026-01-29 20:52:41 +00:00

[PR #505] Fix #502 decoder bug #1150

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#502