41x slower compression speed for 1 pathological file #98

New Issue

claunia · 2026-01-29T20:34:30Z

claunia commented

2026-01-29 20:34:30 +00:00

Originally created by @phlknght on GitHub (Mar 17, 2016).

For one particular file, I am seeing 41x slower compression speed versus the average for my data set, but only on quality 10/11. This file is a 57MB XML file that has the same small section repeated over and over. It compresses down to 13KB, but takes 72 minutes to process. If I reduce the window to 16 bits, it completes in 45 minutes, and at quality 9 with the default window, it compresses in 1.7s. Decompression takes .125s.

For my uses, I will be editing the file to not be so problematic, but there is something pathological about this case that seems to deserve some attention.

Here is a repo containing the problem file, compressed with Brotli.

Here is the raw timing data:

$ time ./bro.exe --input /cygdrive/d/brotli/testFiles/bad.xml --output /cygdrive/d/brotli/testFiles/bad.xml.bro -f -q 9

real    0m1.700s
user    0m1.653s
sys     0m0.046s

$ time ./bro.exe --input /cygdrive/d/brotli/testFiles/bad.xml --output /cygdrive/d/brotli/testFiles/bad.xml.bro -f -q 10 -w 16

real    44m53.220s
user    44m50.720s
sys     0m2.386s

$ time ./bro.exe --input /cygdrive/d/brotli/testFiles/bad.xml --output /cygdrive/d/brotli/testFiles/bad.xml.bro -f -q 10

real    71m55.931s
user    71m51.150s
sys     0m3.525s

$ time ./bro.exe --input /cygdrive/d/brotli/testFiles/bad.xml.bro --output /cygdrive/d/brotli/testFiles/bad.xml.unbro -f -d

real    0m0.125s
user    0m0.061s
sys     0m0.062s

I did some quick profiling on the first few seconds within Visual Studio 2015, and here are the results:

Drilling down:

And further down:

Originally created by @phlknght on GitHub (Mar 17, 2016). For one particular file, I am seeing 41x slower compression speed versus the average for my data set, but only on quality 10/11. This file is a 57MB XML file that has the same small section repeated over and over. It compresses down to 13KB, but takes 72 minutes to process. If I reduce the window to 16 bits, it completes in 45 minutes, and at quality 9 with the default window, it compresses in 1.7s. Decompression takes .125s. For my uses, I will be editing the file to not be so problematic, but there is something pathological about this case that seems to deserve some attention. Here is a [repo](https://github.com/phlknght/brotliBadXml) containing the problem file, compressed with Brotli. Here is the raw timing data: ``` $ time ./bro.exe --input /cygdrive/d/brotli/testFiles/bad.xml --output /cygdrive/d/brotli/testFiles/bad.xml.bro -f -q 9 real 0m1.700s user 0m1.653s sys 0m0.046s $ time ./bro.exe --input /cygdrive/d/brotli/testFiles/bad.xml --output /cygdrive/d/brotli/testFiles/bad.xml.bro -f -q 10 -w 16 real 44m53.220s user 44m50.720s sys 0m2.386s $ time ./bro.exe --input /cygdrive/d/brotli/testFiles/bad.xml --output /cygdrive/d/brotli/testFiles/bad.xml.bro -f -q 10 real 71m55.931s user 71m51.150s sys 0m3.525s $ time ./bro.exe --input /cygdrive/d/brotli/testFiles/bad.xml.bro --output /cygdrive/d/brotli/testFiles/bad.xml.unbro -f -d real 0m0.125s user 0m0.061s sys 0m0.062s ``` I did some quick profiling on the first few seconds within Visual Studio 2015, and here are the results: ![brotliperf](https://cloud.githubusercontent.com/assets/17853859/13858480/81e83232-ec44-11e5-8d27-e29471cd34c1.png) Drilling down: ![brotliperf1](https://cloud.githubusercontent.com/assets/17853859/13858482/84e1bdfa-ec44-11e5-99f3-b56aa39519ce.png) And further down: ![brotliperf2](https://cloud.githubusercontent.com/assets/17853859/13858487/87fcda6a-ec44-11e5-8445-ac68e8e4f258.png)

claunia closed this issue

2026-01-29 20:34:31 +00:00

claunia commented

2026-01-29 20:34:33 +00:00

@eustas commented on GitHub (Jun 21, 2016):

Hello.

Sorry for the late response. Going to investigate soon.

@eustas commented on GitHub (Jun 21, 2016): Hello. Sorry for the late response. Going to investigate soon.

claunia commented

2026-01-29 20:34:35 +00:00

@eustas commented on GitHub (Jun 21, 2016):

Seems that problem is already solved (at least in linux):

$ time ./bro -q 11 -f -i ./bad.xml -o ./bad.xml.br; ls -la bad.xml.br

real    0m3.429s
user    0m3.404s
sys 0m0.024s
-rw------- 1 eustas eng 6045 Jun 21 13:53 bad.xml.br

Going to test under windows soon.

@eustas commented on GitHub (Jun 21, 2016): Seems that problem is already solved (at least in linux): ``` $ time ./bro -q 11 -f -i ./bad.xml -o ./bad.xml.br; ls -la bad.xml.br real 0m3.429s user 0m3.404s sys 0m0.024s -rw------- 1 eustas eng 6045 Jun 21 13:53 bad.xml.br ``` Going to test under windows soon.

claunia commented

2026-01-29 20:34:36 +00:00

@eustas commented on GitHub (Jun 21, 2016):

Fixed in windows as well.

@eustas commented on GitHub (Jun 21, 2016): Fixed in windows as well.

claunia referenced this issue

2026-01-29 20:50:26 +00:00

[PR #98] [MERGED] Align distance code meaning in the brotli encoder. #703

Sign in to join this conversation.