Functions to calculate approximate memory usage needed for compression and decompression #122

New Issue

claunia · 2026-01-29T20:36:10Z

claunia commented

2026-01-29 20:36:10 +00:00

Originally created by @GregSlazinski on GitHub (Jul 3, 2016).

Is it possible to provide such functions?
Such as :
int64 BrotliCompressionMemUsage(int quality, int lgwin, BrotliEncoderMode mode);
int64 BrotliDecompressionMemUsage(int quality, int lgwin, BrotliEncoderMode mode);

In my program, I use multi-threaded compression, by executing multiple compressors on multiple threads.
However I limit the number of simultaneous threads by the number of available memory .

For example if system has 4 GB of RAM.
And each call to compress function would require 1 GB of memory usage.
Based on that info I would know that I can create up to 4 threads.

However I was unable to find any information about memory usage for compressing and decompressing with Brotli.

Thank you

Originally created by @GregSlazinski on GitHub (Jul 3, 2016). Is it possible to provide such functions? Such as : int64 BrotliCompressionMemUsage(int quality, int lgwin, BrotliEncoderMode mode); int64 BrotliDecompressionMemUsage(int quality, int lgwin, BrotliEncoderMode mode); In my program, I use multi-threaded compression, by executing multiple compressors on multiple threads. However I limit the number of simultaneous threads by the number of available memory . For example if system has 4 GB of RAM. And each call to compress function would require 1 GB of memory usage. Based on that info I would know that I can create up to 4 threads. However I was unable to find any information about memory usage for compressing and decompressing with Brotli. Thank you

claunia closed this issue

2026-01-29 20:36:10 +00:00

claunia commented

2026-01-29 20:36:11 +00:00

@eustas commented on GitHub (Jul 28, 2016):

Hello. For decompressor it is easy. For compressor we have only rough estimates (there are about 90 places where memory is allocated).
Going to add method for decompressor soon and for compressor after some code combing...

@eustas commented on GitHub (Jul 28, 2016): Hello. For decompressor it is easy. For compressor we have only rough estimates (there are about 90 places where memory is allocated). Going to add method for decompressor soon and for compressor after some code combing...

claunia commented

2026-01-29 20:36:12 +00:00

@eustas commented on GitHub (Jun 9, 2017):

For decompressor the worst case is 2.6MiB + 1.5 * ringbuffer size. Soon an API for calculating ringbuffer size from the first byte will appear + ability to remove 1.5x multiplier.
As for encoder, we still need more time to build the right model.

@eustas commented on GitHub (Jun 9, 2017): For decompressor the worst case is 2.6MiB + 1.5 * ringbuffer size. Soon an API for calculating ringbuffer size from the first byte will appear + ability to remove 1.5x multiplier. As for encoder, we still need more time to build the right model.

claunia commented

2026-01-29 20:36:13 +00:00

@Pro100AlexHell commented on GitHub (Dec 25, 2017):

Hello! Could you please answer:

Can I use custom allocated buffer for brotli decompression and reuse it, for avoid GC in C# in webgl unity?
Which size of memory is need for decompression? May be some table related to compress level?
What about reuse buffers and memory requirements for case when a browser is decompress itself? (after receive comperssion header: br) Is it ok?

@Pro100AlexHell commented on GitHub (Dec 25, 2017): Hello! Could you please answer: 1) Can I use custom allocated buffer for brotli decompression and reuse it, for avoid GC in C# in webgl unity? 2) Which size of memory is need for decompression? May be some table related to compress level? 3) What about reuse buffers and memory requirements for case when a browser is decompress itself? (after receive comperssion header: br) Is it ok?

claunia commented

2026-01-29 20:36:15 +00:00

@eustas commented on GitHub (Oct 22, 2018):

Yes, if you write custom memory manager; both encoder and decoder could be initialized with custom memory manager, which could work with preallocated arena
In worst case 1.5x LZ window + up to 2.6MiB for Huffman tables. 1.5x multiplier could be eliminated by appropriate decoder option; normally Huffman tables are much smaller; 2.6MiB is a theoretical maximum never seen in reality. Normally those are 1-2 orders smaller.
Sorry, could you rephrase the question, please

@eustas commented on GitHub (Oct 22, 2018): 1) Yes, if you write custom memory manager; both encoder and decoder could be initialized with custom memory manager, which could work with preallocated arena 2) In worst case 1.5x LZ window + up to 2.6MiB for Huffman tables. 1.5x multiplier could be eliminated by appropriate decoder option; normally Huffman tables are much smaller; 2.6MiB is a theoretical maximum never seen in reality. Normally those are 1-2 orders smaller. 3) Sorry, could you rephrase the question, please

claunia commented

2026-01-29 20:36:17 +00:00

@Pro100AlexHell commented on GitHub (Oct 23, 2018):

When the browser receive HTTP header with compression "br", it implement decompression itself. Do you know how good is implementation in firefox and chrome? For example, if my html page need 1000 small files, and all of it is compressed with brotli.. is memory reused? or for each file, the browser create new buffer, 2 MiB (or smaller)? It's browser reated, but I don't even know how to test it.

@Pro100AlexHell commented on GitHub (Oct 23, 2018): 3) When the browser receive HTTP header with compression "br", it implement decompression itself. Do you know how good is implementation in firefox and chrome? For example, if my html page need 1000 small files, and all of it is compressed with brotli.. is memory reused? or for each file, the browser create new buffer, 2 MiB (or smaller)? It's browser reated, but I don't even know how to test it.

claunia commented

2026-01-29 20:36:19 +00:00

@eustas commented on GitHub (Oct 24, 2018):

Both Firefox and Chrome use reference (this) decoder.
Decoder does very few allocations, IIRC less than 7. Chrome uses rather effective memory manager (TCMalloc). Combine those facts, and memory reuse would never be a problem.
Can't share exact stats, but in 50+% of cases memory usage is slightly above 32K; in the majority of cases memory usage is below 0.5MiB. That highly depends on the size of the file; to be more precise, it depends on "window size"; Huffman tables together with other all the other overheads is just a tiny fraction of used memory.

@eustas commented on GitHub (Oct 24, 2018): Both Firefox and Chrome use reference (this) decoder. Decoder does very few allocations, IIRC less than 7. Chrome uses rather effective memory manager (TCMalloc). Combine those facts, and memory reuse would never be a problem. Can't share exact stats, but in 50+% of cases memory usage is slightly above 32K; in the majority of cases memory usage is below 0.5MiB. That highly depends on the size of the file; to be more precise, it depends on "window size"; Huffman tables together with other all the other overheads is just a tiny fraction of used memory.

claunia commented

2026-01-29 20:36:21 +00:00

@eustas commented on GitHub (Jan 6, 2023):

BrotliEncoderEstimatePeakMemoryUsage has been added.
As I've mentioned before - for decoder it is easier, but impossible without knowing the first 2 bytes of input...

@eustas commented on GitHub (Jan 6, 2023): `BrotliEncoderEstimatePeakMemoryUsage` has been added. As I've mentioned before - for decoder it is easier, but impossible without knowing the first 2 bytes of input...

claunia referenced this issue

2026-01-29 20:50:36 +00:00

[PR #122] [CLOSED] Fix vs2010 #731

claunia referenced this issue

2026-01-29 20:50:36 +00:00

[PR #122] Fix vs2010 #735

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#122