Ability to clone/serialize/deserialize state in Compressor and Decompressor #469

New Issue

claunia · 2026-01-29T20:44:23Z

claunia commented

2026-01-29 20:44:23 +00:00

Originally created by @bergkvist on GitHub (Jul 5, 2023).

I have a lot of similar files which I want to compress individually. I have noticed however that I can get a much better compression ratio (typically 5x extra) by first processing some training data with the same compressor/decompressor.

import brotli

training_data: bytes = # ... concatenated bytes from multiple files in the training data set
data: bytes = # ... actual data that we want to compress

# This processing takes time, and needs to be repeated for every new piece of "data"
c = brotli.Compressor()
d = brotli.Decompressor()
c.process(training_data)
d.process(c.flush())
# It would be nice if we can clone the compressor/decompressor state to avoid repeating the computation.

c.process(data)
compressed = c.finish()
assert d.process(compressed) == data

Ideally I would want to be able to do something like this:

import brotli

c0 = brotli.Compressor()
d0 = brotli.Decompressor()
c0.process(training_data)
d0.process(c0.flush())

for data in dataset:
    c = c0.clone()
    d = d0.clone()
    c.process(data)
    compressed = c.finish()
    assert d.process(compressed) == data

I have tried using copy.deepcopy, but this fails with TypeError: cannot pickle 'brotli.Compressor' object.

Originally created by @bergkvist on GitHub (Jul 5, 2023). I have a lot of similar files which I want to compress individually. I have noticed however that I can get a much better compression ratio (typically 5x extra) by first processing some training data with the same compressor/decompressor. ```py import brotli training_data: bytes = # ... concatenated bytes from multiple files in the training data set data: bytes = # ... actual data that we want to compress # This processing takes time, and needs to be repeated for every new piece of "data" c = brotli.Compressor() d = brotli.Decompressor() c.process(training_data) d.process(c.flush()) # It would be nice if we can clone the compressor/decompressor state to avoid repeating the computation. c.process(data) compressed = c.finish() assert d.process(compressed) == data ``` Ideally I would want to be able to do something like this: ```py import brotli c0 = brotli.Compressor() d0 = brotli.Decompressor() c0.process(training_data) d0.process(c0.flush()) for data in dataset: c = c0.clone() d = d0.clone() c.process(data) compressed = c.finish() assert d.process(compressed) == data ``` I have tried using `copy.deepcopy`, but this fails with `TypeError: cannot pickle 'brotli.Compressor' object`.

claunia closed this issue

2026-01-29 20:44:23 +00:00

claunia commented

2026-01-29 20:44:23 +00:00

@eustas commented on GitHub (Jul 6, 2023):

I believe, solution for this is not serializtion/deserialization, but using extra dictionaries. Native and Java API have this feature. Python and Go do not have such bindings yet. Hopefully will add them soon.

@eustas commented on GitHub (Jul 6, 2023): I believe, solution for this is not serializtion/deserialization, but using extra dictionaries. Native and Java API have this feature. Python and Go do not have such bindings yet. Hopefully will add them soon.

claunia commented

2026-01-29 20:44:24 +00:00

@bergkvist commented on GitHub (Jul 31, 2023):

Has support for dictionaries in Python been added now? Since this was closed?

@bergkvist commented on GitHub (Jul 31, 2023): Has support for dictionaries in Python been added now? Since this was closed?

claunia commented

2026-01-29 20:44:24 +00:00

@eustas commented on GitHub (Jul 31, 2023):

Not yet, but we are moving towards that. If you like to have that tracked, lets create an issue that reflects our intention.

@eustas commented on GitHub (Jul 31, 2023): Not yet, but we are moving towards that. If you like to have that tracked, lets create an issue that reflects our intention.

claunia commented

2026-01-29 20:44:25 +00:00

@bergkvist commented on GitHub (Jul 31, 2023):

Okay, I created an issue for that now (https://github.com/google/brotli/issues/1052) - feel free to edit/link PRs/issues as appropriate

@bergkvist commented on GitHub (Jul 31, 2023): Okay, I created an issue for that now (https://github.com/google/brotli/issues/1052) - feel free to edit/link PRs/issues as appropriate

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#469