Custom dictionary causes failed round trip on some input #99

New Issue

claunia · 2026-01-29T20:34:30Z

claunia commented

2026-01-29 20:34:30 +00:00

Originally created by @phlknght on GitHub (Mar 15, 2016).

Using BrotliCompressWithCustomDictionary to compress certain files then decompressing after using BrotliSetCustomDictionary followed by BrotliDecompressStream results in output that doesn't match the original file. However, most files survive the round trip intact.

Here is my compress code which is lightly modified from bro.cc, since it is possible that I am doing something wrong:

void Compress(FILE* fin, FILE* fout, FILE* dict, const string& output_path, int window, int quality)
{
    brotli::BrotliParams params;
    params.lgwin = window;
    params.quality = quality;
    try {
        brotli::BrotliFileIn in(fin, 1 << 16);
        brotli::BrotliFileOut out(fout);

        uint8_t* dictBuf = NULL;
        if (dict != NULL)
        {
            dictBuf = new uint8_t[DICTIONARY_SIZE];
            size_t dictSize = fread(dictBuf, 1, DICTIONARY_SIZE, dict);
            if (!BrotliCompressWithCustomDictionary(dictSize, dictBuf, params, &in, &out)) {
                fprintf(stderr, "compression failed\n");
                _unlink(output_path.c_str());
                exit(1);
            }
        }
        else if (!BrotliCompress(params, &in, &out)) {
            fprintf(stderr, "compression failed\n");
            _unlink(output_path.c_str());
            exit(1);
        }
    } catch (std::bad_alloc&) {
        fprintf(stderr, "not enough memory\n");
        _unlink(output_path.c_str());
        exit(1);
    }
}

And here is my decompress function:

void Decompress(FILE* fin, FILE* fout, FILE* dict) {
    uint8_t* input = new uint8_t[kFileBufferSize];
    uint8_t* output = new uint8_t[kFileBufferSize];
    size_t total_out;
    size_t available_in;
    const uint8_t* next_in;
    size_t available_out = kFileBufferSize;
    uint8_t* next_out = output;
    BrotliResult result = BROTLI_RESULT_NEEDS_MORE_INPUT;
    BrotliState s;
    BrotliStateInit(&s);

    uint8_t* dictBuf = NULL;
    if (dict != NULL)
    {
        dictBuf = new uint8_t[DICTIONARY_SIZE];
        size_t dictSize = fread(dictBuf, 1, DICTIONARY_SIZE, dict);
        BrotliSetCustomDictionary(dictSize, dictBuf, &s);
    }
    while (1) {
        if (result == BROTLI_RESULT_NEEDS_MORE_INPUT) {
            if (feof(fin)) {
                break;
            }
            available_in = fread(input, 1, kFileBufferSize, fin);
            next_in = input;
            if (ferror(fin)) {
                break;
            }
        } else if (result == BROTLI_RESULT_NEEDS_MORE_OUTPUT) {
            fwrite(output, 1, kFileBufferSize, fout);
            if (ferror(fout)) {
                break;
            }
            available_out = kFileBufferSize;
            next_out = output;
        } else {
            break; /* Error or success. */
        }
        result = BrotliDecompressStream(&available_in, &next_in,
            &available_out, &next_out, &total_out, &s);
    }
    if (next_out != output) {
        fwrite(output, 1, next_out - output, fout);
    }
    BrotliStateCleanup(&s);
    delete[] input;
    delete[] output;
    delete[] dictBuf;
    if ((result == BROTLI_RESULT_NEEDS_MORE_OUTPUT) || ferror(fout)) {
        fprintf(stderr, "failed to write output\n");
        exit(1);
    } else if (result != BROTLI_RESULT_SUCCESS) { /* Error or needs more input. */
        fprintf(stderr, "corrupt input\n");
        exit(1);
    }
}

I created a repo with my custom dictionary, dictator.out, which I generated with Cloudflare's dictator.go script and then appended 7 bytes to the end because of the LOAD64 padding, along with broken.bin and works.bin. Broken.bin doesn't survive the round trip intact, and works.bin does.

On the positive side, I am getting a nice boost in compression ratio using the custom dictionary. It went from .336 to .285, though I can't completely trust that figure until the round trip test works for all files.

Originally created by @phlknght on GitHub (Mar 15, 2016). Using BrotliCompressWithCustomDictionary to compress certain files then decompressing after using BrotliSetCustomDictionary followed by BrotliDecompressStream results in output that doesn't match the original file. However, most files survive the round trip intact. Here is my compress code which is lightly modified from bro.cc, since it is possible that I am doing something wrong: ``` void Compress(FILE* fin, FILE* fout, FILE* dict, const string& output_path, int window, int quality) { brotli::BrotliParams params; params.lgwin = window; params.quality = quality; try { brotli::BrotliFileIn in(fin, 1 << 16); brotli::BrotliFileOut out(fout); uint8_t* dictBuf = NULL; if (dict != NULL) { dictBuf = new uint8_t[DICTIONARY_SIZE]; size_t dictSize = fread(dictBuf, 1, DICTIONARY_SIZE, dict); if (!BrotliCompressWithCustomDictionary(dictSize, dictBuf, params, &in, &out)) { fprintf(stderr, "compression failed\n"); _unlink(output_path.c_str()); exit(1); } } else if (!BrotliCompress(params, &in, &out)) { fprintf(stderr, "compression failed\n"); _unlink(output_path.c_str()); exit(1); } } catch (std::bad_alloc&) { fprintf(stderr, "not enough memory\n"); _unlink(output_path.c_str()); exit(1); } } ``` And here is my decompress function: ``` void Decompress(FILE* fin, FILE* fout, FILE* dict) { uint8_t* input = new uint8_t[kFileBufferSize]; uint8_t* output = new uint8_t[kFileBufferSize]; size_t total_out; size_t available_in; const uint8_t* next_in; size_t available_out = kFileBufferSize; uint8_t* next_out = output; BrotliResult result = BROTLI_RESULT_NEEDS_MORE_INPUT; BrotliState s; BrotliStateInit(&s); uint8_t* dictBuf = NULL; if (dict != NULL) { dictBuf = new uint8_t[DICTIONARY_SIZE]; size_t dictSize = fread(dictBuf, 1, DICTIONARY_SIZE, dict); BrotliSetCustomDictionary(dictSize, dictBuf, &s); } while (1) { if (result == BROTLI_RESULT_NEEDS_MORE_INPUT) { if (feof(fin)) { break; } available_in = fread(input, 1, kFileBufferSize, fin); next_in = input; if (ferror(fin)) { break; } } else if (result == BROTLI_RESULT_NEEDS_MORE_OUTPUT) { fwrite(output, 1, kFileBufferSize, fout); if (ferror(fout)) { break; } available_out = kFileBufferSize; next_out = output; } else { break; /* Error or success. */ } result = BrotliDecompressStream(&available_in, &next_in, &available_out, &next_out, &total_out, &s); } if (next_out != output) { fwrite(output, 1, next_out - output, fout); } BrotliStateCleanup(&s); delete[] input; delete[] output; delete[] dictBuf; if ((result == BROTLI_RESULT_NEEDS_MORE_OUTPUT) || ferror(fout)) { fprintf(stderr, "failed to write output\n"); exit(1); } else if (result != BROTLI_RESULT_SUCCESS) { /* Error or needs more input. */ fprintf(stderr, "corrupt input\n"); exit(1); } } ``` I created a repo with my [custom dictionary](https://github.com/phlknght/CustomDictionary), dictator.out, which I generated with Cloudflare's dictator.go script and then appended 7 bytes to the end because of the LOAD64 padding, along with broken.bin and works.bin. Broken.bin doesn't survive the round trip intact, and works.bin does. On the positive side, I am getting a nice boost in compression ratio using the custom dictionary. It went from .336 to .285, though I can't completely trust that figure until the round trip test works for all files.

claunia closed this issue

2026-01-29 20:34:32 +00:00

claunia commented

2026-01-29 20:34:34 +00:00

@phlknght commented on GitHub (Mar 15, 2016):

This commit has the same changes built directly into bro.cc. Probably easier to pull this locally to try it out. It also shows the same behavior.

@phlknght commented on GitHub (Mar 15, 2016): This [commit](https://github.com/phlknght/brotli/commit/ff8680c8588478283eca5f66d56ac5d39ecc4fc1) has the same changes built directly into bro.cc. Probably easier to pull this locally to try it out. It also shows the same behavior.

claunia commented

2026-01-29 20:34:35 +00:00

@eustas commented on GitHub (Mar 15, 2016):

Hello.
Thank you for your report. Going to investigate this tomorrow.

@eustas commented on GitHub (Mar 15, 2016): Hello. Thank you for your report. Going to investigate this tomorrow.

claunia commented

2026-01-29 20:34:37 +00:00

@phlknght commented on GitHub (Mar 15, 2016):

Thanks! I went ahead and created the actual test case using your same framework in this commit.

@phlknght commented on GitHub (Mar 15, 2016): Thanks! I went ahead and created the actual test case using your same framework in this [commit](https://github.com/phlknght/brotli/commit/772903e324f333b22ae1d3ea90e78f26e9228abd).

claunia commented

2026-01-29 20:34:38 +00:00

@eustas commented on GitHub (Mar 16, 2016):

Hello. We have a fix for this. The problem is in decoder. Will publish fix soon.

@eustas commented on GitHub (Mar 16, 2016): Hello. We have a fix for this. The problem is in decoder. Will publish fix soon.

claunia commented

2026-01-29 20:34:41 +00:00

@phlknght commented on GitHub (Mar 16, 2016):

Thanks for the quick response.

@phlknght commented on GitHub (Mar 16, 2016): Thanks for the quick response.

claunia commented

2026-01-29 20:34:43 +00:00