sharpcompress/docs/SEVENZIP_PROFILING.md

7z/LZMA CPU and allocation profile

Scope

Latest comparison on perf/7zip-profile-pooling:

• Baseline: pre-merge f41cc208.
• Updated: merge commit 1ad052ed, incorporating origin/master at 7ddb0121.
• Environment: Release net10.0, .NET 10.0.13, Apple M4/macOS ARM64.

Both versions already contain pooled encoder workspaces, decoder model reuse, stack scratch, direct metadata serialization and pooled LZMA2 compressed output. This comparison measures the upstream merge, not the cumulative improvement over the original unoptimized library.

Outcome: existing allocation savings are preserved. Async archive writing allocates less, especially with large raw headers. Elapsed-time variation does not establish a general throughput improvement or extraction regression.

Measurement method

Libraries and harnesses are saved separately before measurement. No builds or tests run concurrently with measured workloads. Inputs, caller buffers, metadata objects and reflection delegates are prepared outside the measurement window.

• Allocations are exact GC.GetTotalAllocatedBytes(true) deltas, not heap sizes.
• Microbenchmarks use two warmups and five measured iterations; tables show medians.
• Extraction uses two warmups and three measured iterations per process.
• Writer timing repeats alternate version order across three process pairs, giving 15 measured iterations per version after warmup.
• Large-header writing has one untimed validating write, then two warmups and five measured writes. Archive parsing and header CRC checks are outside timing.
• EventPipe sampled-thread-time and allocation traces are collected separately from untraced measurements. On macOS, thread-time samples include waits and safepoints; allocation-tick weights are approximate type attribution.

These are controlled warm-pool allocation measurements, not guarantees of allocation-free cold starts, lower retained capacity, or lower process RSS.

Workloads

Workload	Contents and measurement boundary
Archive extraction	Four 50 MiB files alternating seeded random and structured 64 KiB regions; 200 MiB total output, discarded through a preallocated 64 KiB buffer
Small-file writing	16 independent 4 KiB random files; seekable MemoryStream output, including archive finalization
LZMA2 block writing	Three independently compressed 2 MiB blocks, structured or random; output discarded to Stream.Null
Metadata serialization	10,000 prebuilt entries with long/short Unicode names, partial times, directories and empty files; serializer output discarded
Direct LZMA memory writes	64 writes of the same 4 KiB caller buffer through WriteAsync(ReadOnlyMemory<byte>)
Reset-heavy decoding	64 independently compressed 4 KiB LZMA2 chunks; decoded bytes verified
Large-header archive writing	10,000 empty files with long/short Unicode names and partial times; complete async archive creation with raw or compressed headers

The extraction archive uses solid LZMA2, an 8 MiB dictionary, one compression thread, compression level 3 and compressed headers. It was generated with native 7zz; its compressed size is 106,462,260 bytes. Each extraction opens a new archive over preloaded memory, drains all four entries and checks total output. LeaveStreamOpen = true and EnableParallelism = false are explicit.

Raw entry-stream extraction does not add high-level payload CRC validation. Checksum-validating extraction is tested separately; these timings must not be compared as equivalent work to checksum-validating extraction in other formats. Parallel decoding and a full codec/solidness/I/O/API matrix are outside this profile.

Allocation results

Workload	Baseline allocated	Updated allocated
16-file LZMA writer sync	3,319,696 B	3,319,696 B
16-file LZMA writer async	1,269,480 B	1,187,584 B
16-file LZMA2 writer sync	1,071,896 B	1,071,896 B
16-file LZMA2 writer async	1,148,112 B	1,066,216 B
Three compressible 2 MiB LZMA2 blocks, sync/async	55,144 B	55,144 B
Three random 2 MiB LZMA2 blocks, sync/async	56,704 B	56,704 B
Serialize 10,000 metadata entries	40 B	40 B
64 direct LZMA memory writes	56,408 B	56,408 B
Decode 64 reset chunks sync	712 B	712 B
Decode 64 reset chunks async	8,784 B	8,784 B
200 MiB extraction sync	10,952 B	10,952 B
200 MiB extraction async	26,344 B	26,344 B

Async archive writers save 81,896 B per 16-file batch. Other allocation totals are unchanged. The metadata serializer's remaining 40 bytes are the harness stopwatch; this workload excludes entry construction, header compression and archive finalization, so it is not an allocation-free archive-writing claim.

Complete large-header archive writing

Header	Baseline allocated	Updated allocated	Baseline elapsed	Updated elapsed	Archive bytes
Raw	11,867,624 B	9,164,288 B	7.31 ms	7.47 ms	2,688,632
Compressed	1,076,032 B	994,136 B	142.71 ms	136.91 ms	14,033

Raw-header writing saves 2,703,336 B (23%), primarily by avoiding materialization of pooled header data when calculating CRCs. Both versions produce identical archive lengths; all names parse correctly and header CRC validation succeeds. Identical lengths alone are not a byte-for-byte comparison.

Throughput and CPU observations

The interleaved small-file writer repeat measured:

Writer	Baseline elapsed	Updated elapsed
LZMA sync	22.79 ms	24.55 ms
LZMA async	43.03 ms	43.56 ms
LZMA2 sync	30.55 ms	29.98 ms
LZMA2 async	29.73 ms	30.59 ms

Timings vary in both directions. The extraction comparison also shows execution- order sensitivity:

Extraction comparison	Baseline elapsed	Updated elapsed
Sync, initial run	2.472 s	2.486 s
Async, baseline process first	12.653 s	15.437 s
Async, updated process first	13.600 s	12.714 s

The initial async slowdown did not reproduce when process order was reversed. The reverse-order allocation medians were 26,296 B baseline and 26,344 B updated, only 48 B apart. These results do not establish an extraction speedup or regression.

Separate baseline/updated random-block traces report zero lost events, with four and three allocation ticks respectively. Neither samples recurring per-block compressed byte arrays or encoder optimization workspaces. Both CPU profiles remain dominated by binary-tree matching, encoder block processing and trial decoding. Sparse allocation samples do not support a detailed percentage split; exact allocation deltas are stronger evidence.

Implementation and correctness

The merge retains reusable trial decoders and explicit pooled-model ownership, while adopting upstream's validated/disposable read-only memory adapter and pooled-header CRC handling. Compressed-chunk async copies keep a logical 64 KiB limit rather than requesting 80 KiB and rounding into a 128 KiB pool bucket. Public signatures, dictionary-growth policy, stream ownership defaults and checksum-validation semantics remain unchanged.

• 497 targeted tests and 3,028 complete .NET 10 Release tests pass.
• All six library targets, net48 tests and the performance project compile without warnings/errors. net48 tests were not executed on macOS.
• Coverage includes compressed/uncompressed block boundaries, partial and failed decoding, disposal and buffer returns, property resets and shrinking model banks, non-array memory, cancellation and caller-memory reuse, Unicode names, partial metadata, digest vectors and header CRCs.
• CSharpier and Git whitespace checks pass.

Artifacts and reproduction

Generated archives, library snapshots, harnesses, logs and traces remain outside the repository. The latest comparison is in:

/private/var/folders/6c/j56r6kqn5yj7q1_c6vm5c0t80000gn/T/opencode/sevenzip-profile/round4/

• manifest.json: source revisions and library SHA-256 hashes.
• baseline-library/ and final-library/: separate pre-merge and merged builds.
• micro/, headers/ and the parent directory's harness/: benchmark sources.
• run.py: microbenchmarks, extraction, EventPipe traces and writer timing repeats.
• repeat.py: interleaved small-file writer timing check.
• headers.py: complete raw/compressed-header writing and untimed validation.
• extraction-repeat.py: reverse-order async extraction check.
• *.log and *-analysis.json: raw measurements and trace analysis.

Run scripts from round4/; they use the saved library snapshots. run.py builds each harness before its measurement phase. The parent directory's generate.py creates archive workloads. Open .speedscope.json traces at https://www.speedscope.app/. Tracing uses dotnet-trace 10.0.750501 and the shared TraceEvent 3.1.21 analyzer in rar-100mb-profile/analyzer/.