Custom memory allocation #68

New Issue

claunia · 2026-01-29T20:31:12Z

claunia commented

2026-01-29 20:31:12 +00:00

Originally created by @addaleax on GitHub (Nov 11, 2015).

Hi there!

Many other libraries, especially for compression (including all with zlib-style APIs) expose some way to hand over memory management to the library users, for a variety of reasons. This includes basically all applications that want to have a complete overview over the memory they use, and language bindings which wish to report the memory usage to some kind of garbage collection mechanism, which is the use case where this report is coming from: MayhemYDG/iltorb#3, esp. everything from https://github.com/MayhemYDG/iltorb/issues/3#issuecomment-154582578 downwards.

Now, I have thought about working on a patch for this myself, but support for custom memory allocation would at least have to have a notable impact on the API, and so I don’t really want to start coding anything before a few points are clarified (or before I have convinced you that this would be a useful feature!):

C and C++ API

The decoder obviously should provide some C-only interface for this, e.g. passing a pointer to some structure to functions; The liblzma library supports the following allocator type: lzma_allocator.

This raises the question of whether the encoder should support the same structures: After all, it is written in C++, which has its own Allocator concept, and using an C++-style allocator would have the benefit of being directly applicable to all STL containers.

Still, library users may prefer a consistent API and not have to write different allocator types for encoding and decoding; It would probably be possible to accept the allocator structure used for the decoder and transparently upgrade it to a C++ Allocator type, e.g. by detecting whether it supports template parameters.

Backwards compatibility

I can not tell how important a backwards-compatible interface is right now to you; Allocator arguments would have to be passed to all library entry points, and the C++ side would probably best be served by using template arguments for specifying Allocators (as in the std::vector definition).

This could probably be implemented by making only additions to the API by renaming the old public functions/classes, e.g. renaming BrotliCompressor to BasicBrotliCompressor<…>, adding the changes there, and then defining BrotliCompressor to be a BasicBrotliCompressor<> (imitating the C++ std::basic_string/std::string idea here).

Passing along the allocator structures

I don’t really have an overview of your codebase (yet), but I feel like there’s a lot of functions, esp. in the decoder, which would either have to receive an additional argument, or one would have to add an additional field to BrotliState, and in either way at least the ABI is broken (Side note: If one accepts that fact and goes with adding a field to BrotliState, it might be a good idea to include fields for forward compatibility here, as in e.g. lzma_stream). And anyway, I’m not sure whether BrotliState should contain the allocator methods semantically, since this is not really about the state of the decoder but rather the exact opposite, i.e. about the state of the “outside world”.

So: Is this worthwhile? How would the API best be changed? Let me know what you think!

Originally created by @addaleax on GitHub (Nov 11, 2015). Hi there! Many other libraries, especially for compression (including all with zlib-style APIs) expose some way to hand over memory management to the library users, for a variety of reasons. This includes basically all applications that want to have a complete overview over the memory they use, and language bindings which wish to report the memory usage to some kind of garbage collection mechanism, which is the use case where this report is coming from: MayhemYDG/iltorb#3, esp. everything from https://github.com/MayhemYDG/iltorb/issues/3#issuecomment-154582578 downwards. Now, I have thought about working on a patch for this myself, but support for custom memory allocation would at least have to have a notable impact on the API, and so I don’t really want to start coding anything before a few points are clarified (or before I have convinced you that this would be a useful feature!): #### C and C++ API The decoder obviously should provide some C-only interface for this, e.g. passing a pointer to some structure to functions; The liblzma library supports the following allocator type: [lzma_allocator](https://github.com/addaleax/xz/blob/282e768a1484e88c8b7ec35655ee4959954ec87a/src/liblzma/api/lzma/base.h#L348). This raises the question of whether the encoder should support the same structures: After all, it is written in C++, which has its own [`Allocator`](http://en.cppreference.com/w/cpp/concept/Allocator) concept, and using an C++-style allocator would have the benefit of being directly applicable to all STL containers. Still, library users may prefer a consistent API and not have to write different allocator types for encoding and decoding; It would probably be possible to accept the allocator structure used for the decoder and transparently upgrade it to a C++ Allocator type, e.g. by detecting whether it supports template parameters. #### Backwards compatibility I can not tell how important a backwards-compatible interface is right now to you; Allocator arguments would have to be passed to all library entry points, and the C++ side would probably best be served by using template arguments for specifying Allocators (as in the `std::vector` definition). This could probably be implemented by making only additions to the API by renaming the old public functions/classes, e.g. renaming `BrotliCompressor` to `BasicBrotliCompressor<…>`, adding the changes there, and then defining `BrotliCompressor` to be a `BasicBrotliCompressor<>` (imitating the C++ `std::basic_string`/`std::string` idea here). #### Passing along the allocator structures I don’t really have an overview of your codebase (yet), but I feel like there’s a lot of functions, esp. in the decoder, which would either have to receive an additional argument, or one would have to add an additional field to `BrotliState`, and in either way at least the ABI is broken (Side note: If one accepts that fact and goes with adding a field to `BrotliState`, it might be a good idea to include fields for forward compatibility here, as in e.g. [lzma_stream](https://github.com/addaleax/xz/blob/282e768a1484e88c8b7ec35655ee4959954ec87a/src/liblzma/api/lzma/base.h#L507)). And anyway, I’m not sure whether `BrotliState` should contain the allocator methods semantically, since this is not really about the state of the decoder but rather the exact opposite, i.e. about the state of the “outside world”. So: Is this worthwhile? How would the API best be changed? Let me know what you think!

claunia closed this issue

2026-01-29 20:31:13 +00:00

claunia commented

2026-01-29 20:31:16 +00:00

@addaleax commented on GitHub (Nov 12, 2015):

This would probably also apply to the contained python bindings, as the docs explain:

In most situations, however, it is recommended to allocate memory from the Python heap specifically because the latter is under control of the Python memory manager. For example, this is required when the interpreter is extended with new object types written in C. Another reason for using the Python heap is the desire to inform the Python memory manager about the memory needs of the extension module. Even when the requested memory is used exclusively for internal, highly-specific purposes, delegating all memory requests to the Python memory manager causes the interpreter to have a more accurate image of its memory footprint as a whole. Consequently, under certain circumstances, the Python memory manager may or may not trigger appropriate actions, like garbage collection, memory compaction or other preventive procedures. Note that by using the C library allocator as shown in the previous example, the allocated memory for the I/O buffer escapes completely the Python memory manager.

@addaleax commented on GitHub (Nov 12, 2015): This would probably also apply to the contained python bindings, as the [docs](https://docs.python.org/3/c-api/memory.html) explain: > In most situations, however, it is recommended to allocate memory from the Python heap specifically because the latter is under control of the Python memory manager. For example, this is required when the interpreter is extended with new object types written in C. Another reason for using the Python heap is the desire to inform the Python memory manager about the memory needs of the extension module. Even when the requested memory is used exclusively for internal, highly-specific purposes, delegating all memory requests to the Python memory manager causes the interpreter to have a more accurate image of its memory footprint as a whole. Consequently, under certain circumstances, the Python memory manager may or may not trigger appropriate actions, like garbage collection, memory compaction or other preventive procedures. Note that by using the C library allocator as shown in the previous example, the allocated memory for the I/O buffer escapes completely the Python memory manager.

claunia commented

2026-01-29 20:31:19 +00:00

@eustas commented on GitHub (Nov 16, 2015):

Hello.

Thank you for the interest.
Actually we're going to add custom memory allocation support to decoder soon. Encoder will get it a little bit later.

Also we plan to update our API to converge with API of modern compressors.

Best regards,
Eugene.

@eustas commented on GitHub (Nov 16, 2015): Hello. Thank you for the interest. Actually we're going to add custom memory allocation support to decoder soon. Encoder will get it a little bit later. Also we plan to update our API to converge with API of modern compressors. Best regards, Eugene.

claunia commented

2026-01-29 20:31:20 +00:00

@addaleax commented on GitHub (Nov 16, 2015):

Okay, great! Thanks, and let me know if I can help with anything. :)

@addaleax commented on GitHub (Nov 16, 2015): Okay, great! Thanks, and let me know if I can help with anything. :)

claunia commented

2026-01-29 20:31:21 +00:00

@eustas commented on GitHub (Nov 23, 2015):

Custom memory allocators for decoder feature has been landed recently.

@eustas commented on GitHub (Nov 23, 2015): Custom memory allocators for decoder feature has been landed recently.

claunia commented

2026-01-29 20:31:24 +00:00