starred/brotli
1
0
Fork 0
mirror of https://github.com/google/brotli.git synced 2026-07-08 17:56:58 +00:00
Code Issues 1.2k Packages Projects Releases 20 Wiki Activity

[PR #902] [MERGED] Python: use a new output buffer code #1475

New Issue
Closed
opened 2026-01-29 20:54:18 +00:00 by claunia · 0 comments
claunia commented 2026-01-29 20:54:18 +00:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/google/brotli/pull/902
Author: @ghost
Created: 5/3/2021
Status: ✅ Merged
Merged: 12/29/2022
Merged by: @eustas

Base: master ← Head: blocks_output_buffer_v3

📝 Commits (10+)

📊 Changes

2 files changed (+396 additions, -174 deletions)

View changed files

📝 python/_brotli.c (+372 -142)
📝 setup.py (+24 -32)

📄 Description

Currently, the output buffer is a std::vector<uint8_t>.
When the buffer grows, resizing will cause unnecessary memcpy().

This PR uses a list of bytes object to represent output buffer, can avoid the extra overhead of resizing.
In addition, C++ code can be removed, it's a pure C extension.

Please review the 11 commits one by one.

May 3, 2021 update:

  • Use some code improvements from https://bugs.python.org/issue41486
  • Exactly the same behavior before the patch, easy for review.
  • Refactoring to make the code more concise, easy for maintenance.

Maybe the block sizes can be adjusted to better suit web applications.
Currently, the first block is 32KB, the second block is 64KB, ...

static const Py_ssize_t BUFFER_BLOCK_SIZE[] =
    { 32*KB, 64*KB, 256*KB, 1*MB, 4*MB, 8*MB, 16*MB, 16*MB,
      32*MB, 32*MB, 32*MB, 32*MB, 64*MB, 64*MB, 128*MB, 128*MB,
      256*MB };

Benchmarks:

The first column is output date size, the unit is MB.
The second and third columns are consumed time, in seconds.

size before   after
0    0.00006  0.00001
10   0.02449  0.02165
20   0.04640  0.03691
30   0.06695  0.05128
40   0.08199  0.06622
50   0.10581  0.08062
60   0.12336  0.09513
70   0.15077  0.11034
80   0.16498  0.12552
90   0.17981  0.13893
100  0.21600  0.15383
110  0.22965  0.17016
120  0.24485  0.18513
130  0.26064  0.19902
140  0.30751  0.21165
150  0.32188  0.22795
160  0.34317  0.24160
170  0.35872  0.25647
180  0.36515  0.27271
190  0.38189  0.28756
200  0.39475  0.30176

The first column is output date size, the unit is KB.

size before   after
0    0.00672  0.00669
20   0.00007  0.00011
40   0.00010  0.00013
60   0.00016  0.00017
80   0.00032  0.00024
100  0.00034  0.00028
120  0.00041  0.00032
140  0.00053  0.00037
160  0.00048  0.00043
180  0.00054  0.00057
200  0.00052  0.00053
220  0.00056  0.00071
240  0.00062  0.00062
260  0.00081  0.00071
280  0.00085  0.00090
300  0.00093  0.00090
320  0.00093  0.00093
340  0.00098  0.00091
360  0.00104  0.00106
380  0.00103  0.00142
400  0.00105  0.00114
420  0.00121  0.00116
440  0.00118  0.00128
460  0.00123  0.00137
480  0.00122  0.00178
500  0.00148  0.00142

Benchmark code:

from time import perf_counter
from brotli import *

MB = 1024*1024
KB = 1024

for i in range(0, 200*MB+1, 10*MB):
    dat1 = i * b'a'
    dat2 = compress(dat1)
    
    t1 = perf_counter()
    dat3 = decompress(dat2)
    t2 = perf_counter()
    print(i//MB, '%.5f' % (t2-t1))
    
    assert dat1 == dat3
    
for i in range(0, 500*KB+1, 20*KB):
    dat1 = i * b'a'
    dat2 = compress(dat1)
    
    t1 = perf_counter()
    dat3 = decompress(dat2)
    t2 = perf_counter()
    print(i//KB, '%.5f' % (t2-t1))
    
    assert dat1 == dat3

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/google/brotli/pull/902 **Author:** [@ghost](https://github.com/ghost) **Created:** 5/3/2021 **Status:** ✅ Merged **Merged:** 12/29/2022 **Merged by:** [@eustas](https://github.com/eustas) **Base:** `master` ← **Head:** `blocks_output_buffer_v3` --- ### 📝 Commits (10+) - [`331a24c`](https://github.com/google/brotli/commit/331a24c13ae772fcd7dc31e30673a347754b0613) 1. blocks output buffer - [`baa30fd`](https://github.com/google/brotli/commit/baa30fdaab6f79889543518fae0f24d979b1c288) 2. brotli_decompress - [`d5326ab`](https://github.com/google/brotli/commit/d5326ab563e19215da08299882e1d6f8dd446624) 3. compress_stream - [`f1d04a9`](https://github.com/google/brotli/commit/f1d04a97dad161f1af1a2a1c00c1b3d436ea2f82) 4. brotli_Compressor_process - [`8458127`](https://github.com/google/brotli/commit/84581279654a475440446ba069d9c428beeca656) 5. brotli_Compressor_flush - [`25549fc`](https://github.com/google/brotli/commit/25549fce41a7abd12acb5bd6d72793cac597d9da) 6. brotli_Compressor_finish - [`71eaefb`](https://github.com/google/brotli/commit/71eaefbb118ddb8b2205b52b8c6ed4b885d8d958) 7. decompress_stream - [`d37501c`](https://github.com/google/brotli/commit/d37501cc5563471f991b318da0f7f83d4d346497) 8. brotli_Decompressor_process - [`9755e25`](https://github.com/google/brotli/commit/9755e2574bbfc583c8d1b5aa04166d6d9d622f75) 9. brotli_Decompressor_is_finished - [`5f379e3`](https://github.com/google/brotli/commit/5f379e3f9d1bef1ebdc8f369943ae504642054fc) 10. remove C++ code ### 📊 Changes **2 files changed** (+396 additions, -174 deletions) <details> <summary>View changed files</summary> 📝 `python/_brotli.c` (+372 -142) 📝 `setup.py` (+24 -32) </details> ### 📄 Description Currently, the output buffer is a `std::vector<uint8_t>`. When the buffer grows, resizing will cause unnecessary `memcpy()`. This PR uses a list of bytes object to represent output buffer, can avoid the extra overhead of resizing. In addition, C++ code can be removed, it's a pure C extension. Please review the 11 commits one by one. **May 3, 2021 update**: - Use some code improvements from https://bugs.python.org/issue41486 - Exactly the same behavior before the patch, easy for review. - Refactoring to make the code more concise, easy for maintenance. Maybe the block sizes can be adjusted to better suit web applications. Currently, the first block is 32KB, the second block is 64KB, ... ```c static const Py_ssize_t BUFFER_BLOCK_SIZE[] = { 32*KB, 64*KB, 256*KB, 1*MB, 4*MB, 8*MB, 16*MB, 16*MB, 32*MB, 32*MB, 32*MB, 32*MB, 64*MB, 64*MB, 128*MB, 128*MB, 256*MB }; ```` Benchmarks: ``` The first column is output date size, the unit is MB. The second and third columns are consumed time, in seconds. size before after 0 0.00006 0.00001 10 0.02449 0.02165 20 0.04640 0.03691 30 0.06695 0.05128 40 0.08199 0.06622 50 0.10581 0.08062 60 0.12336 0.09513 70 0.15077 0.11034 80 0.16498 0.12552 90 0.17981 0.13893 100 0.21600 0.15383 110 0.22965 0.17016 120 0.24485 0.18513 130 0.26064 0.19902 140 0.30751 0.21165 150 0.32188 0.22795 160 0.34317 0.24160 170 0.35872 0.25647 180 0.36515 0.27271 190 0.38189 0.28756 200 0.39475 0.30176 ``` ``` The first column is output date size, the unit is KB. size before after 0 0.00672 0.00669 20 0.00007 0.00011 40 0.00010 0.00013 60 0.00016 0.00017 80 0.00032 0.00024 100 0.00034 0.00028 120 0.00041 0.00032 140 0.00053 0.00037 160 0.00048 0.00043 180 0.00054 0.00057 200 0.00052 0.00053 220 0.00056 0.00071 240 0.00062 0.00062 260 0.00081 0.00071 280 0.00085 0.00090 300 0.00093 0.00090 320 0.00093 0.00093 340 0.00098 0.00091 360 0.00104 0.00106 380 0.00103 0.00142 400 0.00105 0.00114 420 0.00121 0.00116 440 0.00118 0.00128 460 0.00123 0.00137 480 0.00122 0.00178 500 0.00148 0.00142 ``` Benchmark code: ```python from time import perf_counter from brotli import * MB = 1024*1024 KB = 1024 for i in range(0, 200*MB+1, 10*MB): dat1 = i * b'a' dat2 = compress(dat1) t1 = perf_counter() dat3 = decompress(dat2) t2 = perf_counter() print(i//MB, '%.5f' % (t2-t1)) assert dat1 == dat3 for i in range(0, 500*KB+1, 20*KB): dat1 = i * b'a' dat2 = compress(dat1) t1 = perf_counter() dat3 = decompress(dat2) t2 = perf_counter() print(i//KB, '%.5f' % (t2-t1)) assert dat1 == dat3 ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
claunia added the pull-request label 2026-01-29 20:54:18 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
feature
help wanted
pull-request
Mirrored from GitHub Pull Request
 release-v1.1
release-v1.1.1
release-v1.2
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/brotli#1475
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?