sharpcompress/docs/ZIP_PROFILING.md

ZIP allocation and throughput profile

Scope

Profiled master at 7ddb0121, using Release net10.0, .NET 10.0.13 and Apple M4/macOS ARM64. The checkout was rebuilt and the library copied before measurement. The initial diagnostic results below are the baseline; the allocation-optimization follow-up records changes and staged measurements separately.

The payload is identical to the RAR case: four 50 MiB files alternating 64 KiB seeded-random and structured regions, 200 MiB extracted. Native 7zz produced two ordinary, unencrypted, single-volume ZIPs:

Compression	Archive bytes	SHA-256
Deflate, level 5, one thread	109,948,058	5e8bbcc810ac6835a39bc9090a56c4ed186b3c0b56bc4c9ae15de01791b829b6
Stored	209,715,758	bebeab9c94dc4ed2fc9c5d6d5c4716b60c94a1e4b0859b5f532cbaabe266a977

Python's ZIP reader independently checked all entry CRCs before profiling. The same archived bytes are used by SharpCompress and System.IO.Compression. This is not a Zip64, encryption, split-archive or ZIP-writing evaluation.

Method

• Each scenario runs in a fresh process, with two warmups and seven measured extractions. Tables show medians for the complete 200 MiB payload.
• SharpCompress Archive and Reader APIs are measured synchronously and asynchronously. Async cases use async open/enumeration/entry-stream APIs. The .NET 10 reference uses ZipArchive.CreateAsync and ZipArchiveEntry.OpenAsync.
• All paths use the same explicit CRC32 pass and verify each entry's CRC, length and the overall entry count. A separate raw control omits CRC computation but still verifies lengths. These are raw entry-stream drains, not high-level extraction-to-directory APIs.
• Memory input and the caller's 64 KiB output buffer are allocated outside the measurement window. Output is discarded. Each extraction opens a new archive.
• File input uses a 4 KiB FileStream buffer with SequentialScan and Asynchronous, including for sync tests. Filesystem caches are warm; these numbers are not cold-disk results.
• LeaveStreamOpen = true and EnableParallelism = false are explicit. The harness verifies the caller's source remains readable after archive disposal.
• Allocation deltas use GC.GetTotalAllocatedBytes(true) around extraction and disposal, including the 40-byte stopwatch but excluding logging and CPU-counter queries. These are managed allocations, not retained memory or process RSS.
• Separate EventPipe traces repeat ten measured extractions after warmup. No builds, tests, RAR profiles or native sampling overlap timed ZIP measurements.

Deflate extraction with CRC validation

Input / API	Allocated bytes	Elapsed
Memory / SharpCompress Archive sync	445,104	241.34 ms
Memory / SharpCompress Reader sync	445,312	244.18 ms
Memory / SharpCompress Archive async	2,127,304	245.39 ms
Memory / SharpCompress Reader async	1,160,112	246.71 ms
Memory / .NET ZipArchive sync	8,816	175.84 ms
Memory / .NET ZipArchive async	10,808	175.38 ms
File / SharpCompress Archive sync	449,640	250.90 ms
File / SharpCompress Reader sync	449,600	254.09 ms
File / SharpCompress Archive async	4,799,624	274.25 ms
File / SharpCompress Reader async	3,135,352	268.42 ms
File / .NET ZipArchive sync	13,104	184.96 ms
File / .NET ZipArchive async	3,884,928	214.04 ms

On memory-backed input, async Archive allocates about 1.68 MB more than sync; async Reader allocates about 0.71 MB more. File-backed async allocations also include suspended state machines. The .NET reference itself allocates substantially more with asynchronous file input, so not all file-backed overhead is specific to SharpCompress.

Stored and checksum controls

Memory-backed API	Deflate without CRC	Stored with CRC	Stored without CRC
SharpCompress Archive sync	163.15 ms	82.70 ms	4.00 ms
SharpCompress Reader sync	165.96 ms	83.91 ms	4.42 ms
SharpCompress Archive async	166.06 ms	102.63 ms	4.36 ms
SharpCompress Reader async	168.62 ms	83.11 ms	4.16 ms
.NET ZipArchive sync	103.18 ms	82.70 ms	3.65 ms
.NET ZipArchive async	101.73 ms	83.15 ms	3.96 ms

The Deflate-only gap persists without the common CRC pass: roughly 163–169 ms versus 102–103 ms. Stored ZIP is much closer, pointing to the codec/read chain rather than a general ZIP-container bottleneck. The stored async Archive CRC timing was higher in this process; it is not evidence of a separate codec cost.

Memory-backed stored extraction allocates only about 8–11 KB per operation. The common CRC pass costs roughly 75–85 ms on this host and dominates stored extraction, so CRC-inclusive timings alone obscure codec performance.

Allocation findings

1. Task-returning compressed-input read chain

Memory-backed allocation samples attribute about 1.7 MB/extraction to Task<int> in async Archive and about 0.7 MB in async Reader. The principal Archive chain is:

ZlibBaseStream.ReadAsync(byte[], ...)
  -> ReadOnlySubStream.ReadAsync(byte[], ...)
  -> SharpCompressStream.ReadAsync(byte[], ...)
  -> SourceStream.ReadAsync(byte[], ...)

src/SharpCompress/Compressors/Deflate/ZlibBaseStream.cs refills its default 16 KiB compressed-input buffer through byte-array/Task overloads, even on modern targets. The wrappers already have Memory/ValueTask overloads, but this call path does not select them. Archive has more wrappers than Reader, explaining much of the allocation difference.

First candidate: carry Memory/ValueTask reads through the compressed-input path and provide a ValueTask-based codec read core. Preserve legacy/generated sync behavior, input limits, stream ownership and cancellation. Savings must be measured; existing Task-returning public overloads still need compatible adapters.

2. Per-block Huffman scratch arrays

Both sync and async traces attribute roughly 0.4 MB/extraction to int[] allocations in InflateBlocks.Process, accounting for most sync allocations. In src/SharpCompress/Compressors/Deflate/Inflate.cs, dynamic-tree construction allocates four single-element arrays (bl, bd, tl, td) for output values. Fixed-tree construction also uses small arrays and array wrappers.

Second candidate: replace these temporary array-shaped outputs with ref/out values, or bounded reusable/stack scratch. Renting a separate pool buffer for each single integer would add overhead rather than address the representation. The large Huffman workspace and sliding window are already pooled; do not count them as fresh per-block allocations. Initialization and corrupt-input behavior must remain intact.

3. Memory reads rent and copy decoded output

ZlibBaseStream.ReadAsync(Memory<byte>) currently rents an array, delegates to the byte-array overload, copies decoded bytes into the caller's memory, and returns the rental on every read. This is mostly a copy/dispatch cost after warmup, not the dominant fresh-array allocation source.

Third candidate: decode directly into array-backed caller memory, retaining a correct pooled fallback for non-array memory. A future core can eliminate the intermediate Task as well. Merely changing callers to the array overload is not an allocation fix: a controlled probe measured:

Async Deflate / memory input	Memory overload	Array overload
Archive allocated	2,127,304 B	2,357,704 B
Reader allocated	1,160,112 B	1,620,912 B

Array-overload probes were slightly faster in some runs but allocated more through additional Task-returning wrappers. Fix the library core rather than recommending this as a caller-side allocation optimization.

Caller-buffer sensitivity

Async Deflate / memory input	16 KiB caller buffer	64 KiB	256 KiB
Archive allocated	2,818,504 B	2,127,304 B	1,954,504 B
Reader allocated	1,851,360 B	1,160,112 B	987,312 B

Larger caller buffers reduce per-output-read Tasks but leave the compressed-input chain in place. Throughput stayed around 245–250 ms; increasing output buffers alone is not a solution to the underlying allocation cost. Caller-buffer storage is outside these allocation deltas and larger buffers have a memory tradeoff.

CPU interpretation and lower-priority work

Managed traces place work in the inflater, InflateFast, CRC and table setup. They also attribute a surprisingly large leaf share to ZeroMemoryInternal under inflater/tree code. A separate macOS native sample did not corroborate a dominant native memset hotspot and could not resolve most JIT frames. Do not interpret that managed leaf percentage as proof that clearing costs half the CPU, or remove required initialization based on it. Exact allocations and untraced timings are the stronger evidence here.

File traces include Monitor waits and suspended read state machines; sampled thread time is not exact instruction CPU time. All nine EventPipe traces report zero lost events. The shared analyzer has too few relevant CPU samples for some .NET-reference traces; that does not mean the reference consumes no CPU.

For four entries, metadata-only opens allocate about 6.4–7.0 KB in SharpCompress and take 16–33 microseconds. The .NET reference takes about 3–4 microseconds. This is negligible beside full extraction, so metadata object pooling is not the first target for this workload.

The next CPU investigation should isolate the managed inflater from stream and output-copy overhead, then measure hot-loop changes with CRC disabled as well as enabled. The .NET reference is a useful native-backed comparison, not a promise that a pure managed implementation can match every timing.

Validation and artifacts

• Every measured CRC-enabled extraction verifies four entry CRCs and 200 MiB of output. Raw controls still verify entry lengths and total output.
• Independent ZIP validation passed for both generated archives.
• 1,002 ZIP/Deflate targeted tests pass on .NET 10 Release.
• The initial investigation made no library changes. Implementation changes are described separately below; staging and commits remain under user control.

Artifacts are outside the repository:

/private/var/folders/6c/j56r6kqn5yj7q1_c6vm5c0t80000gn/T/opencode/zip-200mb-profile/

manifest.json records the commit, archive hashes and library hash. prepare.py creates/validates archives and snapshots the already-built library. run.py reproduces the matrix, controls, buffer sweep and traces. probe.py compares read overloads; native-sample.py performs the separate native-stack cross-check. Harness sources, frozen library, logs, traces and analysis JSON are retained. Tracing uses dotnet-trace 10.0.750501 and the shared TraceEvent analyzer under rar-100mb-profile/analyzer/; its RAR-named phase marker is reused by the ZIP harness and does not imply RAR decoding. Open Speedscope traces at https://www.speedscope.app/.

Allocation-optimization follow-up

The follow-up saves four independent library snapshots before measurement: baseline, reads (target 1), trees (targets 1–2), and direct (targets 1–3). They use the original ZIPs, caller buffers, CRC checks and harness boundaries. Artifacts and the comparison script are in the artifact directory's optimization/ subdirectory; library-hashes.json identifies each snapshot. These exploratory snapshots precede the final generated-sync overload correction. The final snapshot and final.py revalidate the actual retained code; use the verified-* logs for the final before/after comparison, not exploratory timings.

Retained implementation

1. Async read chain: compressed-input refills select Memory/ValueTask overloads on modern targets. A shared ValueTask read core avoids materializing a Task for memory-based output reads. Public byte-array signatures remain unchanged, and generated sync reads retain byte-array I/O. Legacy targets keep their existing byte-array refill path.
2. Huffman outputs: fixed and dynamic tree builders use scalar/ref/out parameters instead of allocating single-element arrays and array wrappers for each block. Existing bit-length initialization and invalid-tree checks remain. The large pooled workspace and sliding window are unchanged.
3. Output memory: array-backed memory decodes directly into the requested array segment. Non-array memory retains the pooled fallback and returns its rental in finally. The codec's output-array reference is cleared after reads, including cancellation/failure, so caller memory or returned rentals are not retained by the decoder between operations.

No public API signatures or format support change. These shared codec changes also require ZIP, GZip, Zlib and legacy-target validation; they do not introduce a new sync Span read override or a new global mutable codec pool.

Allocation savings by stage

For the same memory-backed 200 MiB extraction with CRCs and a 64 KiB caller buffer:

API	Baseline	Target 1	Targets 1–2	All three targets
Archive sync	445,104 B	445,104 B	34,448 B	34,448 B
Reader sync	445,312 B	445,312 B	34,656 B	34,656 B
Archive async	2,127,304 B	445,896 B	35,240 B	35,240 B
Reader async	1,160,112 B	445,872 B	35,216 B	35,216 B

Target 1 removes the recurring Task allocation premium. Target 2 removes 410,656 B/extraction of tree scratch in this workload. Target 3 avoids the per-output-read rental/copy but adds no further warm allocation reduction. The final build reproduces these allocation totals while preserving byte-array I/O for generated sync reads.

Final before/after comparison

Deflate / API	Baseline allocated	Final allocated	Baseline elapsed	Final elapsed
Memory / Archive sync	445,104 B	34,448 B	279.58 ms	273.74 ms
Memory / Reader sync	445,312 B	34,656 B	266.99 ms	259.80 ms
Memory / Archive async	2,127,304 B	35,240 B	287.89 ms	265.48 ms
Memory / Reader async	1,160,112 B	35,216 B	272.96 ms	264.22 ms
File / Archive sync	450,280 B	38,984 B	280.99 ms	277.79 ms
File / Reader sync	449,600 B	38,944 B	268.66 ms	274.04 ms
File / Archive async	4,799,632 B	2,897,656 B	288.79 ms	286.93 ms
File / Reader async	3,133,968 B	2,129,952 B	291.82 ms	302.76 ms

Memory-backed allocation savings are approximately 92% sync, 98% async Archive, and 97% async Reader. File-backed async allocations remain affected by suspension and scheduling: their remaining samples are primarily async state machine boxes rather than the removed per-block arrays and completed-read Tasks.

Two additional process pairs reversed execution order with CRC computation disabled. Medians across their 14 measured extractions per version:

Memory / raw Deflate	Baseline elapsed	Final elapsed
Archive sync	178.85 ms	173.08 ms
Archive async	178.33 ms	173.01 ms
Reader async	182.00 ms	175.60 ms

The repeat suggests a modest 3–4% improvement for these memory-backed cases, not a general throughput guarantee. File results vary in both directions. Target 3's isolated timing benefit is small/variable, so it is retained as copy elimination without claiming a separate universal speedup. These changes do not establish that the original native-backed .NET inflater throughput gap is closed.

Final async allocation totals are unchanged across 16, 64 and 256 KiB caller buffers: 35,240 B Archive / 35,216 B Reader. The former buffer-size-dependent Task allocation cost is gone. Stored controls retain their original allocation totals; the changes target the Deflate decoder.

Final traces and validation

All six exploratory and four final EventPipe traces report zero lost events. The baseline memory-backed Archive trace samples approximately 1.7 MB/extraction of Task<int> and 0.4 MB of int[]. Final memory traces no longer sample the recurring Task sites and have only four/five allocation ticks across ten measured extractions. Some per-codec setup arrays remain; sparse samples and harness JSON initialization do not support a detailed percentage split at this smaller total.

• 1,018 targeted ZIP/Deflate/GZip/Zlib tests and 2,989 complete .NET 10 Release tests pass.
• All six library targets, net48 tests and the performance project compile without warnings/errors. net48 tests are not executed on macOS.
• New tests cover fixed/stored block transitions and read-ahead, empty and oversubscribed trees, sliced array memory, non-array memory, fragmented and genuinely suspended input, cancellation, corrupt input, I/O failures, caller memory reuse and clearing codec output references. A many-read allocation guard checks that the non-array fallback reuses its rental.
• Emitted sync code was inspected. The explicit SYNC_ONLY branch keeps the original Read(byte[], offset, count) input call; a regression source that rejects Span reads protects this behavior.
• CSharpier formatting/check and Git whitespace checks pass.
• No staging or commit is performed by this follow-up.

Ring-buffer follow-up (c78cc86)

The original matrix used seekable memory/file inputs, including for Reader API measurements. It therefore missed the non-seekable ring-buffer path: switching Deflate refills to memory-based reads selected a wrapper implementation that allocated a temporary array for every fresh read and buffered replay.

Commit c78cc86 accesses array-backed memory directly when reading/writing the ring buffer. Non-array memory uses ArrayPool<byte> rentals returned in finally blocks. Underlying reads retain the Memory/ValueTask overload.

Reran the 24-case Deflate/stored, memory/file, SharpCompress/.NET matrix with the same archives, two warmups, seven measured extractions, CRC/size checks and 64 KiB output buffer. Added non-seekable Reader cases by wrapping each input in a forward-only delegate. Each extraction still checks that the borrowed source remains readable. Builds and traces did not overlap timed measurements.

Non-seekable Reader comparison

Both versions use the extended harness. "Before" is the saved pre-fix optimization/final-library snapshot; manifest.json records both DLL hashes. Allocation values are bytes per complete 200 MiB extraction, not retained memory.

Async Reader / input	Before allocated	Fixed allocated	Before elapsed	Fixed elapsed
Deflate / memory	110,149,184 B	35,256 B	249.45 ms	243.41 ms
Deflate / file	114,237,408 B	4,226,200 B	275.71 ms	265.28 ms
Stored / memory	209,808,104 B	10,784 B	94.71 ms	86.58 ms
Stored / file	211,697,560 B	1,943,216 B	161.20 ms	150.70 ms

The removed allocation volume tracks the compressed payload size. Stored reads benefit too because they share the wrapper's memory-based read implementation. File-backed async reads retain suspension/state-machine allocations; the fix is not a claim of allocation-free file extraction. Sync non-seekable allocations are unchanged: 34,696 B / 38,984 B for Deflate memory/file and 10,224 B / 14,512 B for stored memory/file. Timing differences are descriptive single-run medians, not a universal speedup guarantee.

Seekable Deflate rerun

API / input	Fixed allocated	Fixed elapsed
Archive sync / memory	34,448 B	243.22 ms
Reader sync / memory	34,656 B	240.99 ms
Archive async / memory	35,240 B	241.85 ms
Reader async / memory	35,216 B	241.97 ms
Archive sync / file	38,984 B	250.24 ms
Reader sync / file	38,944 B	249.78 ms
Archive async / file	2,898,728 B	262.74 ms
Reader async / file	2,130,928 B	265.69 ms

Seekable-memory allocation totals match the previous final profile exactly. Async memory allocations also remain 35,240 B Archive / 35,216 B Reader across the 16/64/256 KiB output-buffer sweep. The full matrix, raw controls and non-seekable comparisons contain 49 timed scenarios in total.

Three EventPipe traces cover pre-fix/fixed non-seekable async Reader and fixed seekable async Reader. All report zero lost events. The pre-fix non-seekable trace samples approximately 1.10 GB of byte[] over ten measured extractions under SharpCompressStream.ReadWithRingBufferAsync; the fixed non-seekable trace has four allocation ticks and no sampled byte[] hotspot. Exact allocation totals above come from GC counters, not sampled trace sums.

Artifacts are in zip-200mb-profile/ring-buffer-c78cc86/ under the temporary artifact root documented above. run.py reproduces the matrix and traces; summary.json, individual logs, manifest.json, library snapshots and trace analyses preserve the results. Validation before committing the fix: 2,996 .NET 10 Release tests pass, all library targets build without warnings/errors, and CSharpier and Git whitespace checks pass.