Version 0.5.0 expands buffers on OSX #119

New Issue

claunia · 2026-01-29T20:35:45Z

claunia commented

2026-01-29 20:35:45 +00:00

Originally created by @hashbackup on GitHub (Jun 22, 2016).

I'm trying to get Brotli integrated into the backup program I'm developing, HashBackup. Previously I had been testing with a version downloaded May 18th, and it worked fine on OSX and Linux.

I downloaded the latest version today for testing. Instead of compressing, it is apparently expanding. Any clues?

$ py
Python 2.7.9 (default, Jun 22 2016, 12:43:12) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import brotli
>>> x=1024*' '
>>> z=brotli.compress(x)
>>> len(z)
11520
>>> x2=brotli.decompress(z)
>>> x==x2
True
>>>

Here's a test with bro.py:

[jim@mb python]$ py bro.py -i x.bro -o x.cc -d
[jim@mb python]$ ls -l
total 120
-rw-r--r--@ 1 jim  staff    200 Jun 21 07:41 README.md
-rwxr-xr-x@ 1 jim  staff   5440 Jun 21 07:41 bro.py
-rw-r--r--@ 1 jim  staff   9048 Jun 21 07:41 brotlimodule.cc
drwxr-xr-x  6 jim  staff    204 Jun 21 07:41 tests
-rw-r--r--  1 jim  staff  21550 Jun 22 13:12 x.bro
-rw-r--r--  1 jim  staff   9048 Jun 22 13:12 x.cc
[jim@mb python]$ cmp x.cc brotlimodule.cc

Here's a buffer test:

>>> x='jim' + 100*' ' + 'my'
>>> z=brotli.compress(x)
>>> len(z)
10371
>>> z
'\x1bh\x00\x00\x8c\x94n\xe6\xa2lS\x03\xa9\xb4\xbc\x92\x04\xb4@\xf6\x89ree\x00_BrotliBuildHistogramsWithContext\x00_BrotliEstimateBitCostsForLiterals
(a bunch more garbage)
\x00_BrotliEncoderInputBlockSize\x00_BrotliEncoderWriteData\x00enc/././metablock.h\x00_BrotliEncoderCompressStream\x00_BrotliEncoderCompress\x00_BrotliEncoder'
>>> z=z[:200]
>>> brotli.decompress(z)
'jim                                                                                                    my'
>>> len(z)
200
>>> z=z[:100]
>>> brotli.decompress(z)
'jim                                                                                                    my'
>>> z=z[:50]
>>> brotli.decompress(z)
'jim                                                                                                    my'
>>> z
'\x1bh\x00\x00\x8c\x94n\xe6\xa2lS\x03\xa9\xb4\xbc\x92\x04\xb4@\xf6\x89ree\x00_BrotliBuildHistogramsWit'
>>> brotli.decompress(z[:25])
'jim                                                                                                    my'
>>> brotli.decompress(z[:13])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
brotli.error: BrotliDecompress failed
>>> brotli.decompress(z[:20])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
brotli.error: BrotliDecompress failed
>>> brotli.decompress(z[:21])
'jim                                                                                                    my'
>>>

So it appears to be compressing into a temp buffer but not setting the length right. Probably a bug in python/brotlimodule.cc

Originally created by @hashbackup on GitHub (Jun 22, 2016). I'm trying to get Brotli integrated into the backup program I'm developing, HashBackup. Previously I had been testing with a version downloaded May 18th, and it worked fine on OSX and Linux. I downloaded the latest version today for testing. Instead of compressing, it is apparently expanding. Any clues? ``` $ py Python 2.7.9 (default, Jun 22 2016, 12:43:12) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import brotli >>> x=1024*' ' >>> z=brotli.compress(x) >>> len(z) 11520 >>> x2=brotli.decompress(z) >>> x==x2 True >>> ``` Here's a test with bro.py: ``` [jim@mb python]$ py bro.py -i x.bro -o x.cc -d [jim@mb python]$ ls -l total 120 -rw-r--r--@ 1 jim staff 200 Jun 21 07:41 README.md -rwxr-xr-x@ 1 jim staff 5440 Jun 21 07:41 bro.py -rw-r--r--@ 1 jim staff 9048 Jun 21 07:41 brotlimodule.cc drwxr-xr-x 6 jim staff 204 Jun 21 07:41 tests -rw-r--r-- 1 jim staff 21550 Jun 22 13:12 x.bro -rw-r--r-- 1 jim staff 9048 Jun 22 13:12 x.cc [jim@mb python]$ cmp x.cc brotlimodule.cc ``` Here's a buffer test: ``` >>> x='jim' + 100*' ' + 'my' >>> z=brotli.compress(x) >>> len(z) 10371 >>> z '\x1bh\x00\x00\x8c\x94n\xe6\xa2lS\x03\xa9\xb4\xbc\x92\x04\xb4@\xf6\x89ree\x00_BrotliBuildHistogramsWithContext\x00_BrotliEstimateBitCostsForLiterals (a bunch more garbage) \x00_BrotliEncoderInputBlockSize\x00_BrotliEncoderWriteData\x00enc/././metablock.h\x00_BrotliEncoderCompressStream\x00_BrotliEncoderCompress\x00_BrotliEncoder' >>> z=z[:200] >>> brotli.decompress(z) 'jim my' >>> len(z) 200 >>> z=z[:100] >>> brotli.decompress(z) 'jim my' >>> z=z[:50] >>> brotli.decompress(z) 'jim my' >>> z '\x1bh\x00\x00\x8c\x94n\xe6\xa2lS\x03\xa9\xb4\xbc\x92\x04\xb4@\xf6\x89ree\x00_BrotliBuildHistogramsWit' >>> brotli.decompress(z[:25]) 'jim my' >>> brotli.decompress(z[:13]) Traceback (most recent call last): File "<stdin>", line 1, in <module> brotli.error: BrotliDecompress failed >>> brotli.decompress(z[:20]) Traceback (most recent call last): File "<stdin>", line 1, in <module> brotli.error: BrotliDecompress failed >>> brotli.decompress(z[:21]) 'jim my' >>> ``` So it appears to be compressing into a temp buffer but not setting the length right. Probably a bug in python/brotlimodule.cc

claunia closed this issue

2026-01-29 20:35:46 +00:00

claunia commented

2026-01-29 20:35:48 +00:00

@eustas commented on GitHub (Jun 22, 2016):

Going to investigate it tomorrow. Thanks for the heads-up!

@eustas commented on GitHub (Jun 22, 2016): Going to investigate it tomorrow. Thanks for the heads-up!

claunia commented

2026-01-29 20:35:49 +00:00

@hashbackup commented on GitHub (Jun 22, 2016):

Found it: need to subtract available_out:

    ret = PyBytes_FromStringAndSize((char*)output, output_length-available_out);

While you're in there, you might want to bracket your compression, decompression, and string copies with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS.

@hashbackup commented on GitHub (Jun 22, 2016): Found it: need to subtract available_out: ``` ret = PyBytes_FromStringAndSize((char*)output, output_length-available_out); ``` While you're in there, you might want to bracket your compression, decompression, and string copies with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS.

claunia referenced this issue

2026-01-29 20:50:35 +00:00

[PR #119] [MERGED] Deprecate greedy_block_split and enable_context_modeling brotli params. #730

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#119