[PR #901] Python: use a new output buffer code #1478

New Issue

claunia · 2026-01-29T20:54:19Z

claunia commented

2026-01-29 20:54:19 +00:00

Original Pull Request: https://github.com/google/brotli/pull/901

State: closed
Merged: No

Currently, the output buffer is a std::vector<uint8_t>.
When the buffer grows, resizing will cause unnecessary memcpy().

This PR uses a list of bytes object to represent output buffer, can avoid the extra overhead of resizing.
In addition, C++ code can be removed, it's a pure C extension.

Please review the 11 commits one by one.

May 2, 2021 update:

use some code improvements from https://bugs.python.org/issue41486
refactoring to make the code more concise, reducing the chance of mistake during code maintenance

Benchmarks:

The first column is output date size, the unit is MB.
The second and third columns are consumed time, in seconds.

size before   after
0    0.00006  0.00001
10   0.02449  0.02165
20   0.04640  0.03691
30   0.06695  0.05128
40   0.08199  0.06622
50   0.10581  0.08062
60   0.12336  0.09513
70   0.15077  0.11034
80   0.16498  0.12552
90   0.17981  0.13893
100  0.21600  0.15383
110  0.22965  0.17016
120  0.24485  0.18513
130  0.26064  0.19902
140  0.30751  0.21165
150  0.32188  0.22795
160  0.34317  0.24160
170  0.35872  0.25647
180  0.36515  0.27271
190  0.38189  0.28756
200  0.39475  0.30176

The first column is output date size, the unit is KB.

size before   after
0    0.00672  0.00669
20   0.00007  0.00011
40   0.00010  0.00013
60   0.00016  0.00017
80   0.00032  0.00024
100  0.00034  0.00028
120  0.00041  0.00032
140  0.00053  0.00037
160  0.00048  0.00043
180  0.00054  0.00057
200  0.00052  0.00053
220  0.00056  0.00071
240  0.00062  0.00062
260  0.00081  0.00071
280  0.00085  0.00090
300  0.00093  0.00090
320  0.00093  0.00093
340  0.00098  0.00091
360  0.00104  0.00106
380  0.00103  0.00142
400  0.00105  0.00114
420  0.00121  0.00116
440  0.00118  0.00128
460  0.00123  0.00137
480  0.00122  0.00178
500  0.00148  0.00142

Benchmark code:

from time import perf_counter
from brotli import *

MB = 1024*1024
KB = 1024

for i in range(0, 200*MB+1, 10*MB):
    dat1 = i * b'a'
    dat2 = compress(dat1)
    
    t1 = perf_counter()
    dat3 = decompress(dat2)
    t2 = perf_counter()
    print(i//MB, '%.5f' % (t2-t1))
    
    assert dat1 == dat3
    
for i in range(0, 500*KB+1, 20*KB):
    dat1 = i * b'a'
    dat2 = compress(dat1)
    
    t1 = perf_counter()
    dat3 = decompress(dat2)
    t2 = perf_counter()
    print(i//KB, '%.5f' % (t2-t1))
    
    assert dat1 == dat3

**Original Pull Request:** https://github.com/google/brotli/pull/901 **State:** closed **Merged:** No --- Currently, the output buffer is a `std::vector<uint8_t>`. When the buffer grows, resizing will cause unnecessary `memcpy()`. This PR uses a list of bytes object to represent output buffer, can avoid the extra overhead of resizing. In addition, C++ code can be removed, it's a pure C extension. Please review the 11 commits one by one. **May 2, 2021 update**: - use some code improvements from https://bugs.python.org/issue41486 - refactoring to make the code more concise, reducing the chance of mistake during code maintenance Benchmarks: ``` The first column is output date size, the unit is MB. The second and third columns are consumed time, in seconds. size before after 0 0.00006 0.00001 10 0.02449 0.02165 20 0.04640 0.03691 30 0.06695 0.05128 40 0.08199 0.06622 50 0.10581 0.08062 60 0.12336 0.09513 70 0.15077 0.11034 80 0.16498 0.12552 90 0.17981 0.13893 100 0.21600 0.15383 110 0.22965 0.17016 120 0.24485 0.18513 130 0.26064 0.19902 140 0.30751 0.21165 150 0.32188 0.22795 160 0.34317 0.24160 170 0.35872 0.25647 180 0.36515 0.27271 190 0.38189 0.28756 200 0.39475 0.30176 ``` ``` The first column is output date size, the unit is KB. size before after 0 0.00672 0.00669 20 0.00007 0.00011 40 0.00010 0.00013 60 0.00016 0.00017 80 0.00032 0.00024 100 0.00034 0.00028 120 0.00041 0.00032 140 0.00053 0.00037 160 0.00048 0.00043 180 0.00054 0.00057 200 0.00052 0.00053 220 0.00056 0.00071 240 0.00062 0.00062 260 0.00081 0.00071 280 0.00085 0.00090 300 0.00093 0.00090 320 0.00093 0.00093 340 0.00098 0.00091 360 0.00104 0.00106 380 0.00103 0.00142 400 0.00105 0.00114 420 0.00121 0.00116 440 0.00118 0.00128 460 0.00123 0.00137 480 0.00122 0.00178 500 0.00148 0.00142 ``` Benchmark code: ```python from time import perf_counter from brotli import * MB = 1024*1024 KB = 1024 for i in range(0, 200*MB+1, 10*MB): dat1 = i * b'a' dat2 = compress(dat1) t1 = perf_counter() dat3 = decompress(dat2) t2 = perf_counter() print(i//MB, '%.5f' % (t2-t1)) assert dat1 == dat3 for i in range(0, 500*KB+1, 20*KB): dat1 = i * b'a' dat2 = compress(dat1) t1 = perf_counter() dat3 = decompress(dat2) t2 = perf_counter() print(i//KB, '%.5f' % (t2-t1)) assert dat1 == dat3 ```

claunia added the pull-request label 2026-01-29 20:54:19 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#1478