mirror of
https://github.com/google/brotli.git
synced 2026-09-22 06:35:52 +00:00
Limited, fixed-length compression buffer usage with a C interface #79
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @CTrabant on GitHub (Dec 21, 2015).
Here is the use case: a continuous data stream (e.g. sensor data) needs to be compressed into chunked, fixed-length records. Each record needs to be independent, meaning no compression state is shared or otherwise linked to neighbor records.
The current C interface for encoding, specifically BrotliCompressBuffer(), does not allow writing to a limited size buffer, i.e. the output buffer appears to be required to hold the compressed output of the entire input buffer. It is possible that the C++ interface allows this and I cannot figure out how to write a C wrapper for it.
A bit more about my use case: I am assessing whether brotli would be usable as an encoding of data samples in an international standard for time series seismic data. The length of encoded data segments are approximately 448 to 4032 bytes. Any suggestions within this context would be appreciated.
@CTrabant commented on GitHub (Dec 27, 2015):
I found what I was looking for in the form of the zlib-esque streaming interface (using pure C) created by Cloudflare for a NGINX filter module:
https://github.com/cloudflare/ngx_brotli_module
Such an interface included in this reference repository would be very valuable.
P.S. For what it is worth, I also vote for a more traditional "library" approach such done by Cloudflare, in the sense of creating libbrotli.a and providing brotli.h. Continuing the digression, following as many of the points in the Tao of good compression API (http://richg42.blogspot.com/2015/01/good-lossless-codec-api-design.html) will increase usability, increase usage and reduce the re-invention of very common wheels.
@CTrabant commented on GitHub (Dec 28, 2015):
After more exploration it turns out that even with the zlib-esque interface my issue is not solved. While it is possible to compress to a limited length buffer, if the output buffer is not large enough for a full flush it will not be valid compressed data. Bummer.
My quest for a method to "compress as much input as possible into a fixed length buffer (and be independently decompressable)" remains unrealized.
@eustas commented on GitHub (Dec 28, 2015):
Brotli archive header is less than 1 byte -> independently decompressible blocks are almost the same as independent archives.
The coding phase takes not much time, the most of the time is used for finding better data decomposition. Trying to compress blocks of different size in parallel will not add much overhead (comparing to special solution that estimates output size without producing output).
So it looks that writing some wrapper, that tries to find the maximum input block that produces maximal feasible output, is the easiest solution.
If we consider brotli as a format (not implementation) there is a way to create implementation that will be more efficient for specific data type and output size requirements (e.g. it may omit some computations and use precalculated tables). But brotli is a general purpose data format. If you use some specific data format (SEED?) it likely will be more efficient.
@CTrabant commented on GitHub (Dec 30, 2015):
Thanks. The suggested wrapper sounds like a brute force search for the optimal input block size, which would rarely be optimally efficient, but when the blocks are so small perhaps the overhead is minimal. I will give that a try, it will be enough for exploring brotli's potential for this use.
It is for the SEED format that I explore new options. Currently, the allowed encodings only include compression for integer data, floating point data are stored uncompressed. The idea is precisely to find a general compression scheme that a) can be used for any data sample type and b) is broadly supported, will be readable in decades to come and is not seismology specific (and thus benefits from a much larger community). There is certainly a trade-off with efficiency, a scheme specific for the data characteristics would result in better compression.
@CTrabant commented on GitHub (Aug 3, 2016):
Closing for now, I'll wait and see what I can get out of the new pure-C interfaces when a release is cut.