sharpcompress/docs/RAR_PROFILING.md

RAR pooling and async checksum evaluation

Branch and changes

Evaluated on 2026-10-19 against 78006f94 on adam/release-to-master. The comparison uses a clean build of that commit and a separate build containing the working-tree RAR changes, not timings carried over from another branch. The branch already contains the async unpacker-disposal fix.

The RAR changes were subsequently ported to perf/rar-profile-pooling, based directly on master at 0fbf36d6, without the unrelated commits from adam/release-to-master. Measurements below remain explicitly tied to the profiled 78006f94 baseline; they are not a new profile of the master-based port.

Changes:

• Async RAR Archive entry streams now select CRC32 or BLAKE2sp verification using the same rules as the synchronous path. Failed initialization disposes the newly constructed stream before propagating the exception.
• Async RAR5 and RAR2 Huffman-table scratch buffers are lazily rented, reused for the unpacker's lifetime, and returned on disposal. Sync stack buffers are unchanged.
• RAR5 zero-run decoding is bounded by the alphabet size, not a rented array's capacity. RAR2 table-history padding is cleared to preserve the previous zero-initialization behavior.
• CRC32 lookup tables were already cached and shared. They are intentionally not pooled: there is no per-block allocation to eliminate, and returning a table still referenced by checksum streams would be unsafe.

Workload and measurement

The existing synthetic RAR5 workload is reused: 102.50 MiB compressed (107,473,921 bytes), 200 MiB extracted, four 50 MiB files alternating 64 KiB of seeded random data and 64 KiB of structured records. RAR 7.22 created it with normal compression, an 8 MiB dictionary, CRC32, no solid compression, no encryption, and one volume:

rar a -cfg- -m3 -md8m -mt1 -s- -ep -htc -qo- -idq mixed-100mb.rar \
  input/mixed-0.bin input/mixed-1.bin input/mixed-2.bin input/mixed-3.bin

Archive SHA-256: f6dcb78eaffa527f9394f47fc869ee396a180f7cbe6e4eb1430a60fe8b925fdd.

Release net10.0 runs use the same harness and archive for both builds. Each scenario uses a fresh process, two warmups and seven measured extractions. Version order is reversed between the Archive and Reader scenarios. Output is discarded; Archive drains verify extracted byte counts and Reader drains use the normal entry-writing path. Caller-owned streams set LeaveStreamOpen = true.

Memory-backed runs preload compressed input outside the measured region. File-backed runs use a 4 KiB FileStream buffer, SequentialScan, and FileOptions.Asynchronous; their timings include warm filesystem-cache effects, not cold-disk performance. No builds or test runs compete with profiling.

Managed allocations are exact GC.GetTotalAllocatedBytes(true) deltas around extraction and disposal. They exclude input preloading and between-iteration logging but include a 64 KiB caller buffer, unchanged between versions and unused in Reader scenarios. They are allocations, not live-memory measurements. The decoder still needs its pooled 8 MiB dictionary and may rent a 4 MiB temporary output buffer; low warm allocations do not mean those buffers disappeared.

Separate traces perform ten measured extractions after warmup, using dotnet-trace 10.0.750501 with dotnet-sampled-thread-time and GC allocation ticks. The analyzer uses TraceEvent 3.1.21. CPU stack shares are sampled thread time on macOS, not exact instruction CPU time. Inlining can attribute helper work to the decoder loop; file-backed stacks can include waits/safepoints.

Archive async now performs checksum work absent from its baseline. Its timing is therefore not an equivalent-work comparison; Reader async already checked checksums and better isolates the pooling change. This experiment does not justify claims about all RAR files, dictionary sizes, encryption, solid archives, or cold storage.

The recorded runtime is .NET 10.0.13 on the same Apple M4/macOS ARM64 host used for the earlier evaluation.

Fresh results

Medians per extraction of the complete 200 MiB payload:

Input / async API	Baseline elapsed	Updated elapsed	Baseline allocations	Updated allocations
Memory / Archive	702 ms	755 ms	5.09 MiB	1.96 MiB
Memory / Reader	726 ms	764 ms	4.64 MiB	1.51 MiB
File / Archive	716 ms	800 ms	8.37 MiB	5.25 MiB
File / Reader	805 ms	770 ms	11.44 MiB	4.34 MiB

The memory-backed results save approximately 3.13 MiB per extraction, reducing warm allocations by approximately 62% for Archive and 67% for Reader. File-backed allocations include variable async suspension and pool misses. In particular, the file Reader baseline is higher than in the previous branch's run; do not attribute its entire allocation difference to scratch pooling. The memory-backed comparison better isolates that change.

This is not evidence of a CPU speedup. Archive now does additional checksum work, and Reader memory-backed median process CPU was 734 ms before and 770 ms after in this run. Timings vary with JIT code generation and machine state; these short sequential measurements demonstrate allocation savings, not a general throughput improvement.

All six traces report zero lost events. Baseline allocation samples attribute approximately 3.0–3.1 MiB per extraction to ReadTablesAsync arrays; the updated measured phases contain no sampled recurring allocations at those sites. The exact total-allocation deltas above are more precise than the sampled per-type attribution.

The updated memory-backed Archive trace attributes approximately 83.8% of leaf samples to the RAR5 decoder loop, 10.2% to Crc32Stream.CalculateCrc, and 3.4% to MakeDecodeTables. The baseline Archive trace has no bulk CRC hotspot, confirming that checksum work is now present. Remaining memory-backed allocations are primarily Task<int> results; file-backed reads also allocate suspended async state machines and task adapters. Those read-chain allocations are not changed by scratch pooling.

Artifacts and reproduction

Fresh artifacts are isolated from previous branch evaluations:

/private/var/folders/6c/j56r6kqn5yj7q1_c6vm5c0t80000gn/T/opencode/rar-100mb-profile/evaluation-78006f94/

This contains the baseline source export, separate head-harness/ and pooled-harness/ builds, library hashes, repository-state snapshot, RAR diff, untraced logs, raw .nettrace files, .speedscope.json files, and analysis JSON. Open the Speedscope files in https://www.speedscope.app/. Filenames identify version, API, and input source. The original archive and harness are in the parent directory.

From that parent directory, prepare-evaluation.py builds the current HEAD baseline and working-tree harnesses without changing the repository, and evaluate-current.py evaluation-78006f94 runs the comparison. The preparation step assumes the updated library has already been built in Release for net10.0.

Validation

• All 544 targeted RAR tests pass on .NET 10 Release, including CRC32 and BLAKE2sp mismatch detection, both async read overloads, and entry-copy APIs.
• New scratch-buffer tests cover reuse with poisoned buffers, solid entries, oversized rents, RAR2 history padding, cancellation/checksum-failure cleanup, repeated disposal, and non-seekable input ownership.
• The complete .NET 10 Release run on the profiling branch reported 2,995 passed, no failures. Validation of the master-based port is recorded below.
• All six library targets build; net48 test compilation succeeds. net48 tests were not executed on macOS.
• CSharpier formatting/check and git diff --check pass. The RAR source diff matches the saved profiling snapshot.

The master-based port also passes all 544 targeted RAR tests, all six library builds, net48 test compilation, and the CSharpier formatting check. Its complete .NET 10 Release run reports 2,983 passed, no failures.