starred/brotli
1
0
Fork 0
mirror of https://github.com/google/brotli.git synced 2026-07-08 17:56:58 +00:00
Code Issues 1.2k Packages Projects Releases 20 Wiki Activity

[PR #1280] Improve throughput performance at compression level 9 #1858

New Issue
Open
opened 2026-01-29 20:56:10 +00:00 by claunia · 0 comments
claunia commented 2026-01-29 20:56:10 +00:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/google/brotli/pull/1280
Author: @uttampawar
Created: 6/12/2025
Status: 🔄 Open

Base: master ← Head: level9-opt

📝 Commits (1)

  • cf1a2b9 Improve throughput performance at compression level 9 by adjusting M_MMAP_THRESHOLD value.

📊 Changes

1 file changed (+7 additions, -0 deletions)

View changed files

📝 c/enc/encode.c (+7 -0)

📄 Description

This patch specifically addresses issue observed at level 9, it doesn't have adverse impact at other levels.
Following are the performance numbers in terms of throughput (bytes processed per sec) for various inputs on latest Xeon server.

Runtime 10 seconds
Compression level 9

Input input-sz compressed-sz Default - MiB Opt - MiB Opt/Default
x 1 5 0.27 3.62 13.41
xyzzy 5 9 0.61 1.65 2.70
xyzzy.compressed 9 13 0.64 2.58 4.03
64x 64 10 8.73 27.13 3.11
alice29.txt 152089 51054 11.36 27.73 2.44
alice29.txt.compressed 50096 50100 7.13 6.91 0.97
asyoulik.txt 125179 46694 9.85 27.16 2.76
asyoulik.txt.compressed 45687 45691 6.68 6.61 0.99
backward65536 65792 19 2359.99 4371.97 1.85
bb.binast 12356697 5412654 5.89 5.89 1.00
compressed_file 50096 50100 7.13 7.01 0.98
compressed_file.compressed 50100 50104 7.13 7.09 0.99
compressed_repeated 144224 50443 15.90 168.83 10.62
compressed_repeated.compressed 50299 50303 6.90 6.73 0.98
cp1251-utf16le 1554 660 1.32 38.11 28.87
empty.compressed.17 65538 17 2794.64 4558.76 1.63
empty.compressed.18 196610 22 4568.71 6190.96 1.36
lcet10.txt 426754 127437 17.17 26.17 1.52
lcet10.txt.compressed 124719 124724 14.65 14.37 0.98
mapsdatazrh 285886 166978 16.09 30.18 1.88
mapsdatazrh.compressed 161743 161748 18.12 18 0.99
monkey 843 423 1.50 32.68 21.79
plrabn12.txt 481861 177362 14.75 20.08 1.36
plrabn12.txt.compressed 174771 174776 19.93 19.78 0.99
quickfox_repeated 176128 51 2202.14 5649.04 2.57
random_chunks 2704 1906 2.25 44.59 19.82
random_org_10k.bin 10000 10004 3.25 125.47 38.61
random_org_10k.bin.compressed 10004 10008 3.20 119.01 37.19
ukkonooa 119 71 1.48 13.6 9.19
index.html (from cloudfare) 29329 7476 4.76 45.24 9.50
Average gain 7.50x

Background

  • The "perf" profile showed, major time spent in the kernel.

Children Self Shared Object Command

70.32%    70.32%  [unknown]
29.55%    29.55%  libbrotlienc.so.1.1.0
 0.12%     0.12%  libc.so.6
 0.01%     0.01%  bench
 0.00%     0.00%  ld-linux-x86-64.so.2

The detail stack trace shows following,

Children Self Command Shared Object Symbol

99.91%    29.39%  libbrotlienc.so.1.1.0
        |
        |--75.54%--CreateBackwardReferencesNH5
        |          |
        |          |--54.11%--asm_exc_page_fault
        |          |          |
        |          |           --46.60%--exc_page_fault
        |          |                     |
        |          |                     |--45.08%--do_user_addr_fault
        |          |                     |          |
        |          |                     |          |--20.26%--handle_mm_fault
        |          |                     |          |
        |          |                     |           --0.84%--lock_vma_under_rcu
        |          |                     |
        |          |                      --0.59%--irqentry_exit
        |          |
        |          |--7.61%--sync_regs
        |          |
        |          |--0.87%--asm_sysvec_apic_timer_interrupt
        |          |          |
        |          |           --0.87%--sysvec_apic_timer_interrupt
        |          |                     |
        |          |                      --0.70%--__sysvec_apic_timer_interrupt
        |          |
        |           --0.84%--error_entry
        |
        |--12.19%--0
        |          |
        |          |--9.45%--CreateBackwardReferencesNH5
        |          |          |
        |          |           --6.98%--asm_exc_page_fault
        |          |
        |           --2.68%--0x7a3250ffe010
        |                     CreateBackwardReferencesNH5
        |
        |--9.54%--BrotliEncoderDestroyInstance
        |          |
        |           --9.53%--__munmap
        |                     entry_SYSCALL_64_after_hwframe
        |
         --0.54%--0x400
                   |
                    --0.54%--CreateBackwardReferencesNH5

This gave clear indication of major cycles spent due to "page-faults". Collecting "perf stat" showed below stats,
$ perf stat -- ./bench -q 9 -c 1 index.html
Tested file index.html; size: 29329
Threads: 1, alg: brotli, quality 9
Total times compressed: 1716; compressed size: 7476
Compression speed:4.80 MiB

Performance counter stats for './bench -q 9 -c 1 index.html':

     10,005.57 msec task-clock                       #    1.000 CPUs utilized
            21      context-switches                 #    2.099 /sec
             0      cpu-migrations                   #    0.000 /sec
     9,060,486      page-faults                      #  905.544 K/sec <-------- 9 million page-faults
38,884,787,373      cycles                           #    3.886 GHz
64,575,947,425      instructions                     #    1.66  insn per cycle
11,559,990,014      branches                         #    1.155 G/sec
    70,260,354      branch-misses                    #    0.61% of all branches

  10.007755894 seconds time elapsed

   2.164726000 seconds user
   7.841751000 seconds sys

With suggested change page faults dropped considerably improving the performance.

Tested file index.html; size: 29329
Threads: 1, alg: brotli, quality 9
Total times compressed: 16109; compressed size: 7476
Compression speed:45.06 MiB

Performance counter stats for './bench -q 9 -c 1 index.html':

     10,022.94 msec task-clock                       #    1.000 CPUs utilized
            39      context-switches                 #    3.891 /sec
             2      cpu-migrations                   #    0.200 /sec
        24,359      page-faults                      #    2.430 K/sec <-------- reduced page-faults
38,898,170,252      cycles                           #    3.881 GHz
111,210,331,663      instructions                     #    2.86  insn per cycle
22,419,526,620      branches                         #    2.237 G/sec
   528,617,633      branch-misses                    #    2.36% of all branches

  10.027335163 seconds time elapsed

   9.987097000 seconds user
   0.039175000 seconds sys

And majority cycles are spent in the application instead of kernel managing memory (mmap/munmap).

Children Self Shared Object

98.48%    98.48%  libbrotlienc.so.1.1.0
 0.80%     0.80%  [unknown]
 0.63%     0.63%  libc.so.6
 0.04%     0.04%  bench
 0.04%     0.04%  libm.so.6
 0.01%     0.01%  libbrotlicommon.so.1.1.0
 0.01%     0.01%  [vdso]
 0.00%     0.00%  ld-linux-x86-64.so.2

Environment:

OS: Ubuntu 24.04.2 LTS
Kernel: 6.8.0-58-generic
GCC: gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Glibc: ldd (Ubuntu GLIBC 2.39-0ubuntu8.4) 2.39

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/google/brotli/pull/1280 **Author:** [@uttampawar](https://github.com/uttampawar) **Created:** 6/12/2025 **Status:** 🔄 Open **Base:** `master` ← **Head:** `level9-opt` --- ### 📝 Commits (1) - [`cf1a2b9`](https://github.com/google/brotli/commit/cf1a2b949716b48113fa27494d57e9f5b3c613a1) Improve throughput performance at compression level 9 by adjusting M_MMAP_THRESHOLD value. ### 📊 Changes **1 file changed** (+7 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `c/enc/encode.c` (+7 -0) </details> ### 📄 Description This patch specifically addresses issue observed at level 9, it doesn't have adverse impact at other levels. Following are the performance numbers in terms of throughput (bytes processed per sec) for various inputs on latest Xeon server. Runtime 10 seconds Compression level 9 | Input | input-sz | compressed-sz | Default - MiB| Opt - MiB| Opt/Default| | -------- | ------- |----------| ----------| -------| ------| |x|1|5|0.27|3.62| 13.41 | |xyzzy|5|9|0.61|1.65| 2.70 | |xyzzy.compressed|9|13|0.64|2.58| 4.03 | |64x|64|10|8.73|27.13| 3.11 | |alice29.txt|152089|51054|11.36|27.73| 2.44 | |alice29.txt.compressed|50096|50100|7.13|6.91| 0.97 | |asyoulik.txt|125179|46694|9.85|27.16| 2.76 | |asyoulik.txt.compressed|45687|45691|6.68|6.61| 0.99 | |backward65536|65792|19|2359.99|4371.97| 1.85 | |bb.binast|12356697|5412654|5.89|5.89| 1.00 | |compressed_file|50096|50100|7.13|7.01| 0.98 | |compressed_file.compressed|50100|50104|7.13|7.09| 0.99 | |compressed_repeated|144224|50443|15.90|168.83| 10.62 | |compressed_repeated.compressed|50299|50303|6.90|6.73| 0.98 | |cp1251-utf16le|1554|660|1.32|38.11| 28.87 | |empty.compressed.17|65538|17|2794.64|4558.76| 1.63 | |empty.compressed.18|196610|22|4568.71|6190.96| 1.36 | |lcet10.txt|426754|127437|17.17|26.17| 1.52 | |lcet10.txt.compressed|124719|124724|14.65|14.37| 0.98 | |mapsdatazrh|285886|166978|16.09|30.18| 1.88 | |mapsdatazrh.compressed|161743|161748|18.12|18| 0.99 | |monkey|843|423|1.50|32.68| 21.79 | |plrabn12.txt|481861|177362|14.75|20.08| 1.36 | |plrabn12.txt.compressed|174771|174776|19.93|19.78| 0.99 | |quickfox_repeated|176128|51|2202.14|5649.04| 2.57 | |random_chunks|2704|1906|2.25|44.59| 19.82 | |random_org_10k.bin|10000|10004|3.25|125.47| 38.61 | |random_org_10k.bin.compressed|10004|10008|3.20|119.01| 37.19 | |ukkonooa|119|71|1.48|13.6| 9.19 | |index.html (from cloudfare)|29329|7476|4.76|45.24| 9.50 | |Average gain|||||7.50x| ## Background * The "perf" profile showed, major time spent in the kernel. ### Children Self Shared Object Command 70.32% 70.32% [unknown] 29.55% 29.55% libbrotlienc.so.1.1.0 0.12% 0.12% libc.so.6 0.01% 0.01% bench 0.00% 0.00% ld-linux-x86-64.so.2 The detail stack trace shows following, ### Children Self Command Shared Object Symbol 99.91% 29.39% libbrotlienc.so.1.1.0 | |--75.54%--CreateBackwardReferencesNH5 | | | |--54.11%--asm_exc_page_fault | | | | | --46.60%--exc_page_fault | | | | | |--45.08%--do_user_addr_fault | | | | | | | |--20.26%--handle_mm_fault | | | | | | | --0.84%--lock_vma_under_rcu | | | | | --0.59%--irqentry_exit | | | |--7.61%--sync_regs | | | |--0.87%--asm_sysvec_apic_timer_interrupt | | | | | --0.87%--sysvec_apic_timer_interrupt | | | | | --0.70%--__sysvec_apic_timer_interrupt | | | --0.84%--error_entry | |--12.19%--0 | | | |--9.45%--CreateBackwardReferencesNH5 | | | | | --6.98%--asm_exc_page_fault | | | --2.68%--0x7a3250ffe010 | CreateBackwardReferencesNH5 | |--9.54%--BrotliEncoderDestroyInstance | | | --9.53%--__munmap | entry_SYSCALL_64_after_hwframe | --0.54%--0x400 | --0.54%--CreateBackwardReferencesNH5 This gave clear indication of major cycles spent due to "page-faults". Collecting "perf stat" showed below stats, $ perf stat -- ./bench -q 9 -c 1 index.html Tested file index.html; size: 29329 Threads: 1, alg: brotli, quality 9 Total times compressed: 1716; compressed size: 7476 Compression speed:4.80 MiB Performance counter stats for './bench -q 9 -c 1 index.html': 10,005.57 msec task-clock # 1.000 CPUs utilized 21 context-switches # 2.099 /sec 0 cpu-migrations # 0.000 /sec 9,060,486 page-faults # 905.544 K/sec <-------- 9 million page-faults 38,884,787,373 cycles # 3.886 GHz 64,575,947,425 instructions # 1.66 insn per cycle 11,559,990,014 branches # 1.155 G/sec 70,260,354 branch-misses # 0.61% of all branches 10.007755894 seconds time elapsed 2.164726000 seconds user 7.841751000 seconds sys With suggested change page faults dropped considerably improving the performance. Tested file index.html; size: 29329 Threads: 1, alg: brotli, quality 9 Total times compressed: 16109; compressed size: 7476 Compression speed:45.06 MiB Performance counter stats for './bench -q 9 -c 1 index.html': 10,022.94 msec task-clock # 1.000 CPUs utilized 39 context-switches # 3.891 /sec 2 cpu-migrations # 0.200 /sec 24,359 page-faults # 2.430 K/sec <-------- reduced page-faults 38,898,170,252 cycles # 3.881 GHz 111,210,331,663 instructions # 2.86 insn per cycle 22,419,526,620 branches # 2.237 G/sec 528,617,633 branch-misses # 2.36% of all branches 10.027335163 seconds time elapsed 9.987097000 seconds user 0.039175000 seconds sys And majority cycles are spent in the application instead of kernel managing memory (mmap/munmap). ### Children Self Shared Object 98.48% 98.48% libbrotlienc.so.1.1.0 0.80% 0.80% [unknown] 0.63% 0.63% libc.so.6 0.04% 0.04% bench 0.04% 0.04% libm.so.6 0.01% 0.01% libbrotlicommon.so.1.1.0 0.01% 0.01% [vdso] 0.00% 0.00% ld-linux-x86-64.so.2 ## Environment: OS: Ubuntu 24.04.2 LTS Kernel: 6.8.0-58-generic GCC: gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Glibc: ldd (Ubuntu GLIBC 2.39-0ubuntu8.4) 2.39 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
claunia added the pull-request label 2026-01-29 20:56:10 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
feature
help wanted
pull-request
Mirrored from GitHub Pull Request
 release-v1.1
release-v1.1.1
release-v1.2
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/brotli#1858
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?