Performance cliff when using large available_in (regression) #169

New Issue

claunia · 2026-01-29T20:39:13Z

claunia commented

2026-01-29 20:39:13 +00:00

Originally created by @aidanhs on GitHub (May 23, 2017).

Introduced in cdca91b6f5.

When compiling the given test.c file with script.sh (just put them both in the same directory and run script.sh), you should see the following timing differences on compressing a 100MB buffer of zeroes (before and after the commit above):

+ ./b1

real    0m0.563s
user    0m0.560s
sys     0m0.000s
+ ./b2

real    0m2.676s
user    0m2.672s
sys     0m0.000s

This is most obvious when using BrotliEncoderCompress (uncomment the relevant 6 lines in test.c and comment out the rest to verify), but I've managed to reduce the problem to demonstrate it with BrotliEncoderCompressStream.

The cause appears to be the use of a 'large' value of available_in. If you swap the commented partsize line to be 65536 (matching the buffer size used in the bro tool), the performance difference almost vanishes.

test.c

#include <stdlib.h>
#include <sys/types.h>
#include <brotli/encode.h>

#define MIN(x, y) ((x) < (y) ? (x) : (y))

#define SIZE ((size_t)(100*1024*1024))
static uint8_t IN[SIZE] = { 0 };
static uint8_t OUT[SIZE] = { 0 };

int main(int argc, char** argv) {
    //size_t outsize = SIZE;
    //int rc = BrotliEncoderCompress(
    //    6, BROTLI_DEFAULT_WINDOW, BROTLI_DEFAULT_MODE,
    //    SIZE, IN, &outsize, OUT
    //);
    //if (!rc) { abort(); }

    size_t partsize = 65536;
    //size_t partsize = 1024*1024;
    int quality = 6;
    int lgwin = 22;
    BrotliEncoderState* s = BrotliEncoderCreateInstance(0, 0, 0);
    BrotliEncoderSetParameter(s, BROTLI_PARAM_QUALITY, (uint32_t)quality);
    BrotliEncoderSetParameter(s, BROTLI_PARAM_LGWIN, (uint32_t)lgwin);
    size_t insize = SIZE;
    size_t outsize = SIZE;
    const uint8_t *inbuf = &IN[0];
    const uint8_t **inref = &inbuf;
    uint8_t *outbuf = &OUT[0];
    uint8_t **outref = &outbuf;
    while (insize > 0) {
        size_t thisinsize = MIN(partsize, insize);
        insize -= thisinsize;
        int rc = BrotliEncoderCompressStream(s,
            insize != 0 ? BROTLI_OPERATION_PROCESS : BROTLI_OPERATION_FINISH,
            &thisinsize, inref, &outsize, outref, NULL);
        if (!rc) { abort(); }
        if (thisinsize != 0) { abort(); }
    }
    BrotliEncoderDestroyInstance(s);
}

script.sh

#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset
set -o xtrace

D1=brot1
D2=brot2

(git clone https://github.com/google/brotli.git $D1 && cd $D1 && git checkout cdca91b6f~1 && make lib)
(git clone https://github.com/google/brotli.git $D2 && cd $D2 && git checkout cdca91b6f   && make lib)

cc -Wall -Wextra -O2 -Wno-unused-parameter -o b1 -I$D1/include test.c $D1/libbrotli.a -lm
cc -Wall -Wextra -O2 -Wno-unused-parameter -o b2 -I$D2/include test.c $D2/libbrotli.a -lm

time ./b1
time ./b2

Originally created by @aidanhs on GitHub (May 23, 2017). Introduced in cdca91b6f59dd7632985667. When compiling the given `test.c` file with `script.sh` (just put them both in the same directory and run `script.sh`), you should see the following timing differences on compressing a 100MB buffer of zeroes (before and after the commit above): ``` + ./b1 real 0m0.563s user 0m0.560s sys 0m0.000s + ./b2 real 0m2.676s user 0m2.672s sys 0m0.000s ``` This is most obvious when using `BrotliEncoderCompress` (uncomment the relevant 6 lines in `test.c` and comment out the rest to verify), but I've managed to reduce the problem to demonstrate it with `BrotliEncoderCompressStream`. The cause appears to be the use of a 'large' value of `available_in`. If you swap the commented `partsize` line to be `65536` (matching the buffer size used in the `bro` tool), the performance difference almost vanishes. `test.c` ``` #include <stdlib.h> #include <sys/types.h> #include <brotli/encode.h> #define MIN(x, y) ((x) < (y) ? (x) : (y)) #define SIZE ((size_t)(100*1024*1024)) static uint8_t IN[SIZE] = { 0 }; static uint8_t OUT[SIZE] = { 0 }; int main(int argc, char** argv) { //size_t outsize = SIZE; //int rc = BrotliEncoderCompress( // 6, BROTLI_DEFAULT_WINDOW, BROTLI_DEFAULT_MODE, // SIZE, IN, &outsize, OUT //); //if (!rc) { abort(); } size_t partsize = 65536; //size_t partsize = 1024*1024; int quality = 6; int lgwin = 22; BrotliEncoderState* s = BrotliEncoderCreateInstance(0, 0, 0); BrotliEncoderSetParameter(s, BROTLI_PARAM_QUALITY, (uint32_t)quality); BrotliEncoderSetParameter(s, BROTLI_PARAM_LGWIN, (uint32_t)lgwin); size_t insize = SIZE; size_t outsize = SIZE; const uint8_t *inbuf = &IN[0]; const uint8_t **inref = &inbuf; uint8_t *outbuf = &OUT[0]; uint8_t **outref = &outbuf; while (insize > 0) { size_t thisinsize = MIN(partsize, insize); insize -= thisinsize; int rc = BrotliEncoderCompressStream(s, insize != 0 ? BROTLI_OPERATION_PROCESS : BROTLI_OPERATION_FINISH, &thisinsize, inref, &outsize, outref, NULL); if (!rc) { abort(); } if (thisinsize != 0) { abort(); } } BrotliEncoderDestroyInstance(s); } ``` `script.sh` ``` #!/bin/bash set -o errexit set -o pipefail set -o nounset set -o xtrace D1=brot1 D2=brot2 (git clone https://github.com/google/brotli.git $D1 && cd $D1 && git checkout cdca91b6f~1 && make lib) (git clone https://github.com/google/brotli.git $D2 && cd $D2 && git checkout cdca91b6f && make lib) cc -Wall -Wextra -O2 -Wno-unused-parameter -o b1 -I$D1/include test.c $D1/libbrotli.a -lm cc -Wall -Wextra -O2 -Wno-unused-parameter -o b2 -I$D2/include test.c $D2/libbrotli.a -lm time ./b1 time ./b2 ```

claunia closed this issue

2026-01-29 20:39:13 +00:00

claunia commented

2026-01-29 20:39:13 +00:00

@eustas commented on GitHub (May 23, 2017):

Thanks for the heads-up. Going to investigate it deeper in early June.

@eustas commented on GitHub (May 23, 2017): Thanks for the heads-up. Going to investigate it deeper in early June.

claunia commented

2026-01-29 20:39:14 +00:00

@eustas commented on GitHub (Aug 28, 2017):

Slowdown on all-zeroes data will be fixed with #590
For generic data we expect that "1MB+" heuristic provides both faster and denser compression.

@eustas commented on GitHub (Aug 28, 2017): Slowdown on all-zeroes data will be fixed with #590 For generic data we expect that "1MB+" heuristic provides both faster and denser compression.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#169