how to use brotli::BrotliCompressor, the correct/intended way? #108

New Issue

claunia · 2026-01-29T20:35:02Z

claunia commented

2026-01-29 20:35:02 +00:00

Originally created by @nitram509 on GitHub (May 16, 2016).

Abstract

I appreciate that the brotli library offers a class for stream based compression.
In order to use this class as intended, I have some questions, which the sparse header file didn't answer.

I consider myself as a novice C-Programmer and I'm just following the concept of
don't make me think - applied to API design ;-)

General usage

Just from looking at the public method's signatures, I have some expectations.
I would use the BrotliCompression class like this (pseudo-code).

c = new BrotliCompressor(params);
partialBuffer = new buffer(c->input_block_size())    // use max partial buffer size for stream
for partialBuffer in hugeInputBuffer {
    c->CopyInputToRingBuffer(partialBuffer)
    c->WriteBrotliData(isLast=false, doFlush=false, size(partialBuffer), &outputBuffer)
}
c->WriteBrotliData(isLast=true, doFlush=true, 0, &outputBuffer)  // final flush

I wouldn't expect doing anything more, to write a valid brotli stream, which I can send over wire or save to disk.

Questions

Why are the following methods public, and when to use them?
- WriteMetaBlock()
- WriteMetadata()
- FinishStream
- BrotliSetCustomDictionary()
- WriteStreamHeader()
I didn't use them and the compressor works fine (no errors)
- IMHO, If they are mandatory, the compressor should fail fast

Is_Last vs. Do_Flush vs. FinishStream()

I know, when it comes to file IO, what flush() means. But with these three options I'm confused.

Questions

When do I use what?
What are the consequences?
What does Is_Last do?
- What is the consequence of not doing it?
- Is this mandatory or just a hint?
When there is a FinishStream(),
- Why isn't there a StartStream()? (it's all about wording, e.g. have consistent and symmetric naming)

I'm looking forward to a fruitful discussion :-)

Thanks in advance
Martin

Originally created by @nitram509 on GitHub (May 16, 2016). ### Abstract I appreciate that the brotli library offers a class for stream based compression. In order to use this class as intended, I have some questions, which the sparse header file didn't answer. I consider myself as a novice C-Programmer and I'm just following the concept of [don't make me think](https://www.sensible.com/dmmt.html) - applied to API design ;-) ### General usage Just from looking at the public method's signatures, I have some expectations. I would use the BrotliCompression class like this (pseudo-code). ``` c = new BrotliCompressor(params); partialBuffer = new buffer(c->input_block_size()) // use max partial buffer size for stream for partialBuffer in hugeInputBuffer { c->CopyInputToRingBuffer(partialBuffer) c->WriteBrotliData(isLast=false, doFlush=false, size(partialBuffer), &outputBuffer) } c->WriteBrotliData(isLast=true, doFlush=true, 0, &outputBuffer) // final flush ``` I wouldn't expect doing anything more, to write a valid brotli stream, which I can send over wire or save to disk. ##### Questions - Why are the following methods public, and when to use them? - WriteMetaBlock() - WriteMetadata() - FinishStream - BrotliSetCustomDictionary() - WriteStreamHeader() - I didn't use them and the compressor works fine (no errors) - IMHO, If they are mandatory, the compressor should [fail fast](https://en.wikipedia.org/wiki/Fail-fast) ### Is_Last vs. Do_Flush vs. FinishStream() I know, when it comes to file IO, what flush() means. But with these three options I'm confused. ##### Questions - When do I use what? - What are the consequences? - What does Is_Last do? - What is the consequence of not doing it? - Is this mandatory or just a hint? - When there is a FinishStream(), - Why isn't there a StartStream()? (it's all about wording, e.g. have consistent and symmetric naming) I'm looking forward to a fruitful discussion :-) Thanks in advance Martin

claunia closed this issue

2026-01-29 20:35:04 +00:00

claunia commented

2026-01-29 20:35:06 +00:00

@eustas commented on GitHub (May 17, 2016):

Current interface will be deprecated soon.
We will be grateful if you leave some comments on the future interface:

static const int kBrotliMaxWindowBits = 24;
static const int kBrotliMinWindowBits = 10;
static const int kBrotliMinInputBlockBits = 16;
static const int kBrotliMaxInputBlockBits = 24;

typedef enum BrotliEncoderMode {
  /* Default compression mode. The compressor does not know anything in
     advance about the properties of the input. */
  BROTLI_MODE_GENERIC = 0,
  /* Compression mode for UTF-8 format text input. */
  BROTLI_MODE_TEXT = 1,
  /* Compression mode used in WOFF 2.0. */
  BROTLI_MODE_FONT = 2
} BrotliEncoderMode;

typedef enum BrotliEncoderOperation {
  BROTLI_OPERATION_PROCESS = 0,
  /* Request output stream to flush. Performed when input stream is depleted
     and there is enough space in output stream. */
  BROTLI_OPERATION_FLUSH = 1,
  /* Request output stream to finish. Performed when input stream is depleted
     and there is enough space in output stream. */
  BROTLI_OPERATION_FINISH = 2
} BrotliEncoderOperation;

typedef struct BrotliEncoderParams {
  BrotliEncoderMode mode;
  /* Controls the compression-speed vs compression-density tradeoffs. The higher
     the |quality|, the slower the compression. Range is 0 to 11. */
  int quality;
  /* Base 2 logarithm of the sliding window size. Range is 10 to 24. */
  int lgwin;
  /* Base 2 logarithm of the maximum input block size. Range is 16 to 24.
     If set to 0, the value will be set based on the quality. */
  int lgblock;
} BrotliEncoderParams;

void BrotliEncoderParamsSetDefault(BrotliEncoderParams* params);

/* A state, can not be reused for multiple brotli streams. */
typedef struct BrotliEncoderStateStruct BrotliEncoderState;

/* Fills the new state with a dictionary for LZ77, warming up the ringbuffer,
   e.g. for custom static dictionaries for data formats.
   Not to be confused with the built-in transformable dictionary of Brotli.
   To decode, use BrotliSetCustomDictionary() of the decoder with the same
   dictionary. */
void BrotliEncoderSetCustomDictionary(BrotliEncoderState* state, size_t size,
                                      const uint8_t* dict);

/* Compresses the data in |input_buffer| into |encoded_buffer|, and sets
   |*encoded_size| to the compressed length.
   Returns 0 if there was an error and 1 otherwise. */
int BrotliEncoderCompressBuffer(const BrotliEncoderParams* params,
                                size_t input_size, const uint8_t* input_buffer,
                                size_t* encoded_size, uint8_t* encoded_buffer);

/* Progressively compress input stream and push produced bytes to output stream.
   Workflow consists of 3 tasks:
    * (optional) copy input data to internal buffer
    * actually compress data and (optionally) store it to internal buffer
    * (optional) copy compressed bytes from internal buffer to output stream
   Whenever all 3 tasks can't move forward anymore, or error occurs, this
   method returns.

   |available_in| and |next_in| represent input stream; when X bytes of input
   are consumed, X is subtracted from |available_in| and added to |next_in|.
   |available_out| and |next_out| represent output stream; when Y bytes are
   pushed to output, Y is subtracted from |available_out| and added to
   |next_out|. |total_out| is assigned to the total amount of bytes pushed by
   the instance of encoder to output.

   |op| is used to perform flush or finish the stream.

   Flushing the stream means forcing encoding of all input passed to encoder and
   completing the current output block, so it could be fully decoded by stream
   decoder. To perform flush |op| must be set to BROTLI_OPERATION_FLUSH. Under
   some circumstances (e.g. lack of output stream capacity) this operation would
   require several calls to BrotliEncoderCompressStream. The method must be
   called again until both input stream is depleted and encoder has no more
   output (see BrotliEncoderHasMoreOutput) after the method is called.

   Finishing the stream means encoding of all input passed to encoder and
   adding specific "final" marks, so stream decoder could determine that stream
   is complete. To perform finish |op| must be set to BROTLI_OPERATION_FINISH.
   Under some circumstances (e.g. lack of output stream capacity) this operation
   would require several calls to BrotliEncoderCompressStream. The method must
   be called again until both input stream is depleted and encoder has no more
   output (see BrotliEncoderHasMoreOutput) after the method is called.

   WARNING: when flushing and finishing, |op| should not change until operation
   is complete; input stream should not be refilled as well.

   Returns 0 if there was an error and 1 otherwise.
*/
int BrotliEncoderCompressStream(BrotliEncoderState* s,
                                BrotliEncoderOperation op, size_t* available_in,
                                const uint8_t** next_in, size_t* available_out,
                                uint8_t** next_out, size_t* total_out);

/* Check if encoder is in "finished" state, i.e. no more input is acceptable and
   no more output will be produced.
   Works only with BrotliEncoderCompressStream workflow.
   Returns 1 if stream is finished and 0 otherwise. */
int BrotliEncoderIsFinished(BrotliEncoderState* s);

/* Check if encoder has more output bytes in internal buffer.
   Works only with BrotliEncoderCompressStream workflow.
   Returns 1 if has more output (in internal buffer) and 0 otherwise. */
int BrotliEncoderHasMoreOutput(BrotliEncoderState* s);

@eustas commented on GitHub (May 17, 2016): Current interface will be deprecated soon. We will be grateful if you leave some comments on the future interface: ``` static const int kBrotliMaxWindowBits = 24; static const int kBrotliMinWindowBits = 10; static const int kBrotliMinInputBlockBits = 16; static const int kBrotliMaxInputBlockBits = 24; typedef enum BrotliEncoderMode { /* Default compression mode. The compressor does not know anything in advance about the properties of the input. */ BROTLI_MODE_GENERIC = 0, /* Compression mode for UTF-8 format text input. */ BROTLI_MODE_TEXT = 1, /* Compression mode used in WOFF 2.0. */ BROTLI_MODE_FONT = 2 } BrotliEncoderMode; typedef enum BrotliEncoderOperation { BROTLI_OPERATION_PROCESS = 0, /* Request output stream to flush. Performed when input stream is depleted and there is enough space in output stream. */ BROTLI_OPERATION_FLUSH = 1, /* Request output stream to finish. Performed when input stream is depleted and there is enough space in output stream. */ BROTLI_OPERATION_FINISH = 2 } BrotliEncoderOperation; typedef struct BrotliEncoderParams { BrotliEncoderMode mode; /* Controls the compression-speed vs compression-density tradeoffs. The higher the |quality|, the slower the compression. Range is 0 to 11. */ int quality; /* Base 2 logarithm of the sliding window size. Range is 10 to 24. */ int lgwin; /* Base 2 logarithm of the maximum input block size. Range is 16 to 24. If set to 0, the value will be set based on the quality. */ int lgblock; } BrotliEncoderParams; void BrotliEncoderParamsSetDefault(BrotliEncoderParams* params); /* A state, can not be reused for multiple brotli streams. */ typedef struct BrotliEncoderStateStruct BrotliEncoderState; /* Fills the new state with a dictionary for LZ77, warming up the ringbuffer, e.g. for custom static dictionaries for data formats. Not to be confused with the built-in transformable dictionary of Brotli. To decode, use BrotliSetCustomDictionary() of the decoder with the same dictionary. */ void BrotliEncoderSetCustomDictionary(BrotliEncoderState* state, size_t size, const uint8_t* dict); /* Compresses the data in |input_buffer| into |encoded_buffer|, and sets |*encoded_size| to the compressed length. Returns 0 if there was an error and 1 otherwise. */ int BrotliEncoderCompressBuffer(const BrotliEncoderParams* params, size_t input_size, const uint8_t* input_buffer, size_t* encoded_size, uint8_t* encoded_buffer); /* Progressively compress input stream and push produced bytes to output stream. Workflow consists of 3 tasks: * (optional) copy input data to internal buffer * actually compress data and (optionally) store it to internal buffer * (optional) copy compressed bytes from internal buffer to output stream Whenever all 3 tasks can't move forward anymore, or error occurs, this method returns. |available_in| and |next_in| represent input stream; when X bytes of input are consumed, X is subtracted from |available_in| and added to |next_in|. |available_out| and |next_out| represent output stream; when Y bytes are pushed to output, Y is subtracted from |available_out| and added to |next_out|. |total_out| is assigned to the total amount of bytes pushed by the instance of encoder to output. |op| is used to perform flush or finish the stream. Flushing the stream means forcing encoding of all input passed to encoder and completing the current output block, so it could be fully decoded by stream decoder. To perform flush |op| must be set to BROTLI_OPERATION_FLUSH. Under some circumstances (e.g. lack of output stream capacity) this operation would require several calls to BrotliEncoderCompressStream. The method must be called again until both input stream is depleted and encoder has no more output (see BrotliEncoderHasMoreOutput) after the method is called. Finishing the stream means encoding of all input passed to encoder and adding specific "final" marks, so stream decoder could determine that stream is complete. To perform finish |op| must be set to BROTLI_OPERATION_FINISH. Under some circumstances (e.g. lack of output stream capacity) this operation would require several calls to BrotliEncoderCompressStream. The method must be called again until both input stream is depleted and encoder has no more output (see BrotliEncoderHasMoreOutput) after the method is called. WARNING: when flushing and finishing, |op| should not change until operation is complete; input stream should not be refilled as well. Returns 0 if there was an error and 1 otherwise. */ int BrotliEncoderCompressStream(BrotliEncoderState* s, BrotliEncoderOperation op, size_t* available_in, const uint8_t** next_in, size_t* available_out, uint8_t** next_out, size_t* total_out); /* Check if encoder is in "finished" state, i.e. no more input is acceptable and no more output will be produced. Works only with BrotliEncoderCompressStream workflow. Returns 1 if stream is finished and 0 otherwise. */ int BrotliEncoderIsFinished(BrotliEncoderState* s); /* Check if encoder has more output bytes in internal buffer. Works only with BrotliEncoderCompressStream workflow. Returns 1 if has more output (in internal buffer) and 0 otherwise. */ int BrotliEncoderHasMoreOutput(BrotliEncoderState* s); ```

claunia commented

2026-01-29 20:35:10 +00:00

@nitram509 commented on GitHub (May 29, 2016):

Hi Eugene,

thank you for this outlook.

These are my thoughts/feedback:

stream encoderno externalizes it's state (BrotliEncoderState)
- sadly is no more a class
- but consistent with decoder now
- consistency I prefer more than a state I need to hold
docs of BrotliEncoderCompressStream: ...Workflow consists of 3 tasks...
- this is not clear to me: internal or external workflow
- I suggest to revise the wording to: ... internally workflow consists of 3 tasks...
prefixing constants with 'k' is old school
- I know, this follows Google's C++ guide, nevertheless I think this is awkward
documentation in general is more clear
- e.g. the breadcrumb/hint to custom dictionary from decoder is very helpful.
One thing I miss: How would I initiate a new/empty BrotliEncoderState?

@nitram509 commented on GitHub (May 29, 2016): Hi Eugene, thank you for this outlook. These are my thoughts/feedback: - stream encoderno externalizes it's state (BrotliEncoderState) - sadly is no more a class - but consistent with decoder now - consistency I prefer more than a state I need to hold - docs of BrotliEncoderCompressStream: ...Workflow consists of 3 tasks... - this is not clear to me: internal or external workflow - I suggest to revise the wording to: ... internally workflow consists of 3 tasks... - prefixing constants with 'k' is old school - I know, this follows Google's C++ guide, nevertheless I think this is awkward - documentation in general is more clear - e.g. the breadcrumb/hint to custom dictionary from decoder is very helpful. - One thing I miss: How would I initiate a new/empty BrotliEncoderState?

claunia commented

2026-01-29 20:35:12 +00:00

@nitram509 commented on GitHub (May 29, 2016):

About the comments, what do you think about switching to
Doxygen?

As a benefit for users, you could then automatically generate docs and e.g. upload them to
https://readthedocs.org/.

What do you think?

@nitram509 commented on GitHub (May 29, 2016): About the comments, what do you think about switching to [Doxygen](http://www.stack.nl/~dimitri/doxygen/index.html)? As a benefit for users, you could then automatically generate docs and e.g. upload them to https://readthedocs.org/. What do you think?

claunia commented

2026-01-29 20:35:14 +00:00

@eustas commented on GitHub (Aug 22, 2016):

I have plans to use Doxygen generator, but not sure when it will be integrated to TravisCI automation...

@eustas commented on GitHub (Aug 22, 2016): I have plans to use Doxygen generator, but not sure when it will be integrated to TravisCI automation...

claunia commented

2026-01-29 20:35:16 +00:00

@nitram509 commented on GitHub (Aug 22, 2016):

In the current v0.5.2 release, there is still a confusing declaration between

bool FinishStream(...);
bool WriteBrotliData(const bool is_last, ...);

@eustas would you please give me a hint, when to use what?

@nitram509 commented on GitHub (Aug 22, 2016): In the current v0.5.2 release, there is still a confusing declaration between - bool FinishStream(...); - bool WriteBrotliData(const bool is_last, ...); @eustas would you please give me a hint, when to use what?

claunia commented

2026-01-29 20:35:18 +00:00

@eustas commented on GitHub (Jun 9, 2017):

Doxygenized API can be found here: https://brotli.org/

@eustas commented on GitHub (Jun 9, 2017): Doxygenized API can be found here: https://brotli.org/

claunia referenced this issue

2026-01-29 20:50:31 +00:00

[PR #108] [MERGED] Use the same hasher for text and font mode. #717

Sign in to join this conversation.