BrotliEncoderSetCustomDictionary() causes "false" OOM #154

New Issue

claunia · 2026-01-29T20:38:56Z

claunia commented

2026-01-29 20:38:56 +00:00

Originally created by @ralfjunker on GitHub (Jan 31, 2017).

In the code below, BrotliEncoderSetCustomDictionary() causes a "false" OOM on compression, which causes the application to halt. Curiously, the OOM is not a real OOM. It only appears as such as a result of malloc(0) which returns NULL.

Without the BrotliEncoderSetCustomDictionary() call all works fine.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include "brotli/encode.h"

int main()
{
  unsigned int i;
  uint8_t dictionary [0x1000];
  uint8_t originalBuffer [0x2000];
  uint8_t compressedBuffer [0x2000];
  size_t availIn, availOut;
  const uint8_t * nextIn;
  uint8_t *nextOut;
  BrotliEncoderState *s;
  BROTLI_BOOL r;

  /* Create random dictionary. */
  for (i = 0; i < sizeof(dictionary); i++) {
    dictionary[i] = rand() % 256;
  }

  /* Create an input buffer exactly two times the dictionary. */
  memmove(originalBuffer, dictionary, sizeof(dictionary));
  memmove(&originalBuffer[0x1000], dictionary, sizeof(dictionary));

  s = BrotliEncoderCreateInstance(NULL, NULL, NULL);
  if (!s)
    printf("Error creating Brotli encoder\n");

  /* Set the  custom dictionary. This causes the OOM during compression. */
  BrotliEncoderSetCustomDictionary(s, sizeof(dictionary), &dictionary);

  availIn = sizeof(originalBuffer);
  nextIn = originalBuffer;

  availOut = sizeof(compressedBuffer);
  nextOut = compressedBuffer;

  /* Because the OOM exits the application, this call does not return. */
  r = BrotliEncoderCompressStream(s,
    BROTLI_OPERATION_FINISH, &availIn, &nextIn, &availOut, &nextOut, NULL);
  if (r != BROTLI_TRUE)
    printf("Error compressing data\n");

  BrotliEncoderDestroyInstance(s);

  return 0;
}

The OOM happens in BrotliSplitBlock(), file block_splitter.c on line 137:

https://github.com/google/brotli/blob/master/enc/block_splitter.c#L137

I hope you can reproduce the issue. If you have questions, please let me know.

Originally created by @ralfjunker on GitHub (Jan 31, 2017). In the code below, `BrotliEncoderSetCustomDictionary()` causes a "false" OOM on compression, which causes the application to halt. Curiously, the OOM is not a real OOM. It only appears as such as a result of `malloc(0)` which returns `NULL`. Without the `BrotliEncoderSetCustomDictionary()` call all works fine. ```C #include <stdio.h> #include <stdlib.h> #include <string.h> #include "brotli/encode.h" int main() { unsigned int i; uint8_t dictionary [0x1000]; uint8_t originalBuffer [0x2000]; uint8_t compressedBuffer [0x2000]; size_t availIn, availOut; const uint8_t * nextIn; uint8_t *nextOut; BrotliEncoderState *s; BROTLI_BOOL r; /* Create random dictionary. */ for (i = 0; i < sizeof(dictionary); i++) { dictionary[i] = rand() % 256; } /* Create an input buffer exactly two times the dictionary. */ memmove(originalBuffer, dictionary, sizeof(dictionary)); memmove(&originalBuffer[0x1000], dictionary, sizeof(dictionary)); s = BrotliEncoderCreateInstance(NULL, NULL, NULL); if (!s) printf("Error creating Brotli encoder\n"); /* Set the custom dictionary. This causes the OOM during compression. */ BrotliEncoderSetCustomDictionary(s, sizeof(dictionary), &dictionary); availIn = sizeof(originalBuffer); nextIn = originalBuffer; availOut = sizeof(compressedBuffer); nextOut = compressedBuffer; /* Because the OOM exits the application, this call does not return. */ r = BrotliEncoderCompressStream(s, BROTLI_OPERATION_FINISH, &availIn, &nextIn, &availOut, &nextOut, NULL); if (r != BROTLI_TRUE) printf("Error compressing data\n"); BrotliEncoderDestroyInstance(s); return 0; } ``` The OOM happens in `BrotliSplitBlock()`, file` block_splitter.c` on line 137: https://github.com/google/brotli/blob/master/enc/block_splitter.c#L137 I hope you can reproduce the issue. If you have questions, please let me know.

claunia closed this issue

2026-01-29 20:38:56 +00:00

claunia commented

2026-01-29 20:38:56 +00:00

@eustas commented on GitHub (Feb 1, 2017):

Hello.

Thanks for the report, going to fix it ASAP.

NB: exit on OOM is considered normal for code in production; use -DBROTLI_ENCODER_CLEANUP_ON_OOM compiler option to allow encoder finish its work and cleanup after itself.

@eustas commented on GitHub (Feb 1, 2017): Hello. Thanks for the report, going to fix it ASAP. *NB:* exit on OOM is considered normal for code in production; use `-DBROTLI_ENCODER_CLEANUP_ON_OOM` compiler option to allow encoder finish its work and cleanup after itself.

claunia commented

2026-01-29 20:38:56 +00:00

@ralfjunker commented on GitHub (Feb 1, 2017):

You are correct in saying that -DBROTLI_ENCODER_CLEANUP_ON_OOM works around the exit. However, compression still fails and BrotliEncoderCompressStream() returns BROTLI_FALSE.

Here is a solution I found to pass my tests: Don't call BROTLI_ALLOC if literals_count is 0. It's a small change to https://github.com/google/brotli/blob/master/enc/block_splitter.c#L137, just replace

uint8_t* literals = BROTLI_ALLOC(m, uint8_t, literals_count);

with

uint8_t* literals = literals_count ? BROTLI_ALLOC(m, uint8_t, literals_count) : NULL;

@ralfjunker commented on GitHub (Feb 1, 2017): You are correct in saying that `-DBROTLI_ENCODER_CLEANUP_ON_OOM` works around the exit. However, compression still fails and `BrotliEncoderCompressStream()` returns `BROTLI_FALSE`. Here is a solution I found to pass my tests: _Don't_ call `BROTLI_ALLOC` if `literals_count` is `0`. It's a small change to https://github.com/google/brotli/blob/master/enc/block_splitter.c#L137, just replace ```C uint8_t* literals = BROTLI_ALLOC(m, uint8_t, literals_count); ``` with ```C uint8_t* literals = literals_count ? BROTLI_ALLOC(m, uint8_t, literals_count) : NULL; ```

claunia referenced this issue

2026-01-29 20:50:45 +00:00

[PR #154] [MERGED] Clarifications to the spec regarding when the stream should be rejected as invalid. #761

claunia referenced this issue

2026-01-29 20:50:47 +00:00

[PR #154] Clarifications to the spec regarding when the stream should be rejected as invalid. #764

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#154