Slow for many small string operations? #66

New Issue

claunia · 2026-01-29T20:31:11Z

claunia commented

2026-01-29 20:31:11 +00:00

Originally created by @leidegre on GitHub (Nov 11, 2015).

I've done some tests with Brotli and Zlib

I have an input file with about 30,000 strings (one per line). Each line is between 100-250 characters.

I want to compress these short strings as fast as possible. I use the BrotliCompressBuffer function.

With quality 0, text mode, I get about 6 K operations per second. That is, Brotli is giving me about 6000 compressed strings per second. The compression ratio is between 1.1-1.3.

For reference, Zlib (fast mode) gets me 94 K operations per second with a compression ratio between 1.0-1.2.

Is this expected? Brotli seems to be quite a bit slower here. The marginal better compression ratio isn't worth the trade off for me right now.

I have another benchmark where I generate low entropy text strings and counter the bytes going in and out (compressed bytes). With Brotli I get about 21 MiB in and 2 MiB out per second, 5.8 K op/s. Ratio around 11. With Zlib I get about 690 MiB in and 151 MiB out, 190 K op/s. Ratio around 4.5.

Despite Brotli having a more than 2x compression ratio here it's taking way too long to set up? From the http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf paper we can see listings of throughput speeds that outclass Zlib. However, I can clearly demonstrate that his is not the case here. Is the algorithm old (I was using the 0.2 release) or slow in this regard, or does it simply have a higher overhead over Zlib and this is why it takes so much longer to do many small compression operations? I noticed that the BrotliCompressor does heap allocations, how much of an issue is that? Is it possible to get a compressor that does not allocate additional memory?

Originally created by @leidegre on GitHub (Nov 11, 2015). I've done some tests with Brotli and Zlib I have an input file with about 30,000 strings (one per line). Each line is between 100-250 characters. I want to compress these short strings as fast as possible. I use the `BrotliCompressBuffer` function. With quality 0, text mode, I get about 6 K operations per second. That is, Brotli is giving me about 6000 compressed strings per second. The compression ratio is between 1.1-1.3. For reference, Zlib (fast mode) gets me 94 K operations per second with a compression ratio between 1.0-1.2. Is this expected? Brotli seems to be quite a bit slower here. The marginal better compression ratio isn't worth the trade off for me right now. I have another benchmark where I generate low entropy text strings and counter the bytes going in and out (compressed bytes). With Brotli I get about 21 MiB in and 2 MiB out per second, 5.8 K op/s. Ratio around 11. With Zlib I get about 690 MiB in and 151 MiB out, 190 K op/s. Ratio around 4.5. Despite Brotli having a more than 2x compression ratio here it's taking way too long to set up? From the http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf paper we can see listings of throughput speeds that outclass Zlib. However, I can clearly demonstrate that his is not the case here. Is the algorithm old (I was using the 0.2 release) or slow in this regard, or does it simply have a higher overhead over Zlib and this is why it takes so much longer to do many small compression operations? I noticed that the `BrotliCompressor` does heap allocations, how much of an issue is that? Is it possible to get a compressor that does not allocate additional memory?

claunia commented

2026-01-29 20:31:13 +00:00

@leidegre commented on GitHub (Jan 22, 2016):

I did both. It was consistently slower in terms of throughput or not effective enough in terms of compression. I can write up a small program that you can use to verify.

Sent from my iPhone

On 21 Jan 2016, at 16:31, Simon Heather notifications@github.com wrote:

Did you change the default quality settings in the Brotli parameters? The default of 11 gives the best compression but is much much slower - try changing to a value of 9 or lower.

—
Reply to this email directly or view it on GitHub.

@leidegre commented on GitHub (Jan 22, 2016): I did both. It was consistently slower in terms of throughput or not effective enough in terms of compression. I can write up a small program that you can use to verify. Sent from my iPhone > On 21 Jan 2016, at 16:31, Simon Heather notifications@github.com wrote: > > Did you change the default quality settings in the Brotli parameters? The default of 11 gives the best compression but is much much slower - try changing to a value of 9 or lower. > > — > Reply to this email directly or view it on GitHub.

claunia commented

2026-01-29 20:31:16 +00:00

@eustas commented on GitHub (Jun 20, 2023):

Even level 9 is uses quite sophisticated techniques. For such small strings I'd recommend to benchmark all compression levels. Then it will be more clears what features help best, and perhaps we will be able to add level / parameter that will be the best fit for the use case.

@eustas commented on GitHub (Jun 20, 2023): Even level 9 is uses quite sophisticated techniques. For such small strings I'd recommend to benchmark all compression levels. Then it will be more clears what features help best, and perhaps we will be able to add level / parameter that will be the best fit for the use case.

claunia referenced this issue

2026-01-29 20:50:06 +00:00

[PR #66] [MERGED] Add the current version of the brotli specification. #661

claunia referenced this issue

2026-01-29 20:50:15 +00:00

[PR #66] Add the current version of the brotli specification. #666

Sign in to join this conversation.