# XZ BCJ pooling and decoder profiling ## Branch and changes Evaluated on 2026-10-24 on `fix/xz-bcj-streaming`. The baseline is `0aaf9b92` (the streaming BCJ correctness fix); the pooled version is `5c46363f`. Both libraries are clean Release builds in separate source exports, using the same harness and inputs. The original broken BCJ implementation is not a meaningful performance baseline: it can omit conversions or corrupt output, so it would not perform equivalent work. Changes evaluated: - The shared 8 KiB BCJ buffer is rented from `ArrayPool.Shared` and returned exactly once on disposal. Processing is bounded by the logical buffer size and valid input count, not an oversized rent's capacity. - All six XZ executable filters use that buffer: x86, ARM, ARM-Thumb, PowerPC, SPARC, and IA-64. Unprocessed instruction tails remain buffered between reads. - XZ blocks retain their filter objects for cleanup. Advancing to another block disposes the previous block, and disposing an XZ stream releases its current block. This returns the BCJ buffer and the LZMA2 decoder's existing pooled dictionary/range buffers; previously the filter chain was not disposed. - Filters dispose their own wrappers, not caller-owned input streams. The LZMA2 wrapper explicitly leaves its compressed input open. Disposed readers reject further reads before they can use or rent buffers again. No range-coder, LZMA decoding, checksum, or async scheduling optimization is included in the pooled comparison. The saved library diff was checked against these commits byte-for-byte. The exported baseline DLL SHA-256 is `f7c9c5ac700315205268a0f1aec0c8517289e6f81bfd9d1459d0ec3c17d5985c`; the pooled DLL SHA-256 is `ab7cee7f00bcc5e37f967ca0f74ff3c34a8eb93ee128cf08ca41488987106961`. ## Workloads and measurement ### Complete XZ decoding The controlled workload is **16.36 MiB compressed** (17,159,680 bytes), **32 MiB decoded** (33,554,432 bytes), in one XZ block using x86 BCJ, LZMA2, an 8 MiB dictionary, and CRC64. Python 3.9.6's liblzma-backed `lzma.compress` created it with preset 6. A generator seeded with `1471` alternates 64 KiB of random bytes and 64 KiB of structured x86 records. Each 16-byte structured record contains CALL and JMP instructions targeting fixed absolute addresses. This exercises both incompressible data and BCJ-friendly instructions. Compressed SHA-256: `d1b182144055a3d1767944f32081bc2142f703372dae9d28f5073b629a733e1b`. Decoded SHA-256: `9150fb1e85094a06b7aeb197908a30d3a8466e4d298f629d13b3f0ca99243715`. Every extraction drains through the XZ checksum and verifies the decoded byte count. Memory-backed scenarios preload compressed bytes outside measurement. File-backed scenarios use a 4 KiB `FileStream` buffer, `SequentialScan`, and `FileOptions.Asynchronous`. Filesystem-cache effects are warm, not cold-storage performance. The synchronous API, async array API (`Task`), and async memory API (`ValueTask`) all use 64 KiB requests with a nonzero destination offset. Each version/scenario runs in a fresh process with two warmups and seven measured extractions. Version order alternates between scenarios. Release `net10.0` runs disable tiered compilation. No builds or tests compete with the timed runs. Measurements use .NET 10.0.13 on Apple M4/macOS 27.0.1 ARM64. Allocation figures are exact process-wide `GC.GetTotalAllocatedBytes(true)` deltas around extraction **and disposal**. They include the caller buffer and stream construction, but exclude compressed-input preloading, process startup, and between-iteration logging. They are allocation counts, not live memory or peak working set. The pooled decoder still needs an 8 MiB dictionary and an 8 KiB BCJ buffer while decoding. ### Isolated executable filters Each of the six filters separately decodes 1 MiB of repeated architecture-specific encoded instructions, without LZMA2 or XZ checksum work. Request sizes are 1, 16, and 10,240 bytes, using sync and both async read overloads. Input setup is excluded; the destination buffer and filter lifetime are included. These artificial inputs isolate wrapper/transform costs and do not represent typical executable files. ### Sampling traces Separate traces use `dotnet-trace` 10.0.750501 with `dotnet-sampled-thread-time`, CLR GC allocation ticks, and custom measured-phase markers. XZ traces contain three extractions after two warmups. Each isolated-filter trace contains 10,000 iterations after warmup. TraceEvent 3.1.21 performs analysis; startup and warmup events are excluded using the phase markers. Stack shares are sampled managed thread time, not exact CPU instruction counts. The report separates stacks containing SharpCompress work from unrelated runtime background stacks. File-backed stacks can include waits/safepoints, and inlining can attribute helper costs to their caller. Allocation ticks provide sampled site attribution; the untraced exact allocation deltas are the authoritative totals. ## Fresh results Medians per complete **32 MiB** XZ extraction, including construction and disposal: | Input / API | Baseline ms | Pooled ms | Baseline allocated bytes | Pooled allocated bytes | | --- | ---: | ---: | ---: | ---: | | Memory / sync | 943.8 | 869.3 | 8,482,480 | 69,584 | | Memory / async array | 6,806.8 | 6,718.0 | 8,952,504 | 523,184 | | Memory / async memory | 6,607.1 | 6,770.7 | 8,952,504 | 523,184 | | File / sync | 1,128.0 | 898.2 | 8,486,704 | 74,096 | | File / async array | 7,892.0 | 7,885.2 | 15,116,192 | 8,695,936 | | File / async memory | 7,899.9 | 17,081.7 | 14,392,056 | 10,287,064 | The memory-backed comparison saves approximately **8 MiB per extraction**: **99.2% of warm sync allocations** and **94.2% of warm async allocations**. Most of this comes from returning the existing pooled LZMA2 dictionary; the BCJ buffer alone accounts for only about 8 KiB. Pooling the BCJ buffer without fixing filter-chain disposal would not return either lease reliably. This is **not evidence of a general throughput improvement**. Memory-backed async process CPU medians were 6,508 ms before versus 6,520 ms after for the array API, and 6,466 ms versus 6,518 ms for the memory API. The file/memory-async case was substantially slower after the change in this run (process CPU medians 7,748 ms versus 10,064 ms, with additional elapsed waiting time). That result is retained, not discarded. File-backed scheduling, suspension allocations, warm pool behavior, and machine state can vary; the smaller file allocation reductions should not be treated as precise attribution to a specific buffer. A separate repeatability/async-I/O investigation is needed before claiming file throughput benefits or explaining that slowdown. ### All six isolated filters Synchronous 1 MiB drains with **10,240-byte reads**, including the destination buffer and construction/disposal: | Filter | Baseline ms | Pooled ms | Baseline allocated bytes | Pooled allocated bytes | | --- | ---: | ---: | ---: | ---: | | x86 | 0.387 | 0.388 | 19,064 | 10,528 | | ARM | 0.614 | 0.607 | 18,992 | 11,128 | | ARM-Thumb | 0.422 | 0.427 | 18,992 | 11,128 | | PowerPC | 0.751 | 0.721 | 18,992 | 11,128 | | SPARC | 0.763 | 0.817 | 18,992 | 11,128 | | IA-64 | 0.325 | 0.328 | 18,992 | 11,128 | The async memory API similarly saves about 8 KiB across all six architectures; the non-x86 cases allocate roughly 19,064 bytes before versus 11,200 bytes after (SPARC's baseline median was 18,776). Their pooled elapsed medians range from 0.327 ms for IA-64 to 0.798 ms for SPARC. The async array path still allocates roughly 26 KiB per drain because its non-cached integer results require tasks. Full per-architecture/per-overload results and all seven samples are in `results.json`; these short microbenchmarks do not establish a CPU speedup. ### Small-read overhead remains Isolated x86, **16-byte requests**, again per 1 MiB: | API | Baseline allocated bytes | Pooled allocated bytes | Baseline ms | Pooled ms | | --- | ---: | ---: | ---: | ---: | | Sync | 8,504 | 304 | 0.660 | 0.743 | | Async array | 9,446,240 | 9,438,376 | 1.709 | 1.796 | | Async memory | 9,056 | 1,192 | 3.437 | 3.436 | The array API's approximately 9 MiB remains after pooling: each 16-byte result is outside the runtime's small integer task cache and passes through multiple `Task` wrappers. By contrast, 1-byte results can use cached tasks. The memory path has low allocations but substantial call/state-machine overhead for tiny reads; low allocations do not imply it is faster than the array path. ### CPU and sampled allocation attribution All **18 traces** contain one complete measured phase and report **zero lost events**. The pooled memory-backed sync trace attributes 82.6% of workload leaf samples to `Decoder.CodeFastCore`, 8.8% to CRC64, 4.4% to the fast matched-literal helper, and 2.1% to x86 BCJ conversion. The corresponding baseline shares are 83.2%, 8.4%, 4.8%, and 2.0%: pooling does not move the main CPU bottleneck. The pooled memory-backed async trace instead attributes 30.6% of workload leaf samples to literal `DecodeNormalAsync`, 16.7% to `BitDecoder.DecodeAsync`, 12.6% to async builder `Start`, and 11.3% to `Normalize2Async`. These per-symbol async operations dominate despite the input being in memory. The corresponding baseline distribution is similar. There is no optimized `CodeFastCore` hotspot on this path. The isolated pooled-filter traces attribute 88.4–94.5% of workload leaf samples to the architecture's converter (x86 91.5%, ARM 93.6%, ARM-Thumb 92.4%, PowerPC 94.4%, SPARC 94.5%, IA-64 88.4%). Buffer copying accounts for roughly 4–8%. These are instruction-heavy synthetic inputs; their rankings are not a cross-architecture throughput prediction for real executables. Baseline memory-backed XZ allocation ticks attribute approximately 24 MiB across three measured extractions to dictionary arrays at `OutWindow.EnsureCapacity`. The pooled memory-backed measured phases contain no sampled allocations at that site; remaining async samples principally identify `Task` results in the read chain. All six pooled isolated-filter traces similarly contain no sampled recurring BCJ-buffer allocations at `SetBaseStream`; their output buffers are still allocated by the harness. Per-site sampling weights are approximate and must not be substituted for the exact allocation table above. File-backed traces still contain dictionary pool misses and suspended async state-machine allocations. Most file-backed thread-time leaf samples are in `Monitor.Enter_Slowpath`; these sampled stacks include waits/safepoints and do **not** demonstrate that a lock consumes 98% of instruction CPU time. They do not explain the adverse elapsed result by themselves, nor justify a lock rewrite. ## Remaining opportunities These describe the profiled pooling baseline; completed follow-ups are recorded separately below rather than rewriting the historical measurements. - Async LZMA2 currently calls the per-symbol async decoder, whereas sync decoding can use the optimized range-decoder path. A future experiment should separate asynchronous input refills from CPU-only decoding. It must not replace async I/O with blocking reads, weaken cancellation, overread a chunk, or bypass corruption checks. This evaluation does not make that change. - Small async array reads can allocate a `Task` at multiple wrapper layers for a non-cached read result. Pooling an 8 KiB buffer cannot remove that cost. Pure delegation and a ValueTask-based internal read chain are separate candidates, with exception/cancellation semantics requiring regression coverage. - Calling the memory overload on `XZStream` does not make its whole chain allocation-free: the stream still uses its array-based async implementation. The isolated BCJ memory path and complete XZ memory path must be measured separately rather than assuming the overload name guarantees equivalent costs. - Checksum lookup tables are shared, not per-block scratch allocations. There is no evidence here to justify pooling them. Checksum work must remain enabled in future throughput comparisons. ## Follow-up: remove duplicate BCJ result tasks On `perf/xz-async-decoding`, the first follow-up keeps the internal BCJ read as a `ValueTask` and creates a task only at the existing public array boundary. Already-buffered reads complete without entering an async state machine. Custom subclasses installing a synchronously throwing input stream retain the previous faulted/canceled-task behavior; sync signatures are unchanged. A fresh before/after comparison against the pooled baseline uses the same isolated 1 MiB x86 workload, 16-byte array reads, two warmups, seven measured drains, Release .NET 10, and disabled tiered compilation. Median allocations fall from **9,438,376 to 4,719,784 bytes**; median elapsed time is **1.695 to 0.963 ms**. This is a small-read wrapper result, not a complete XZ throughput claim. Logs and the follow-up harness are in the parent artifact directory as `fix1-before.log`, `fix1-after.log`, and `fix1-harness/`. All **728 targeted Release tests** pass, including allocation guards for all six architectures and fault/cancellation regressions. All six library targets build, and net48 test compilation passes. The remaining public `Task` result cannot be removed without changing that API; a native memory-based XZ chain remains a separate follow-up. ## Follow-up: native XZ memory read chain The next change adds native memory/ValueTask reads to `XZStream` and `XZBlock` on modern targets, carries memory-based I/O through BCJ refills, and completes already-buffered reads without traversing multiple async wrappers. All buffered bytes still update the block check exactly once. Array API signatures and sync array-based I/O remain unchanged. Native/custom destination memory works with all four XZ check types; SHA-256 uses bounded pooled scratch when required. A fresh comparison against the first follow-up (`de7a5d9`) uses the built-in 1 MiB x86+LZMA2 fixture, 16-byte memory reads at offset 5, two warmups, seven measured complete drains, Release .NET 10, and disabled tiered compilation. Median allocations fall from **14,180,048 to 15,056 bytes**, with elapsed medians **22.399 to 19.422 ms**. Both builds retain checksum verification and return the expected output length. This does not yet address the async range-decoder CPU bottleneck or establish file-backed throughput improvements. Artifacts are `fix2-before.log`, `fix2-after.log`, and `fix2-harness/` in the parent directory. The harness's before/after DLLs were explicitly bound and hash-checked after observing that MSBuild reused an earlier reference when switching HintPath properties; `fix2-harness/comparison.json` records the actual measured hashes. All **764 targeted Release tests** pass, including yielding async-only input, seekable/forward-only streams, native/custom memory, mixed async overloads, bounded allocations, empty/canceled reads, and checksum corruption. All six library targets and net48 test compilation pass. The issue sample still matches liblzma in all 24 read-size/overload/offset combinations. ## Follow-up: bounded async LZMA2 fast decoding The third follow-up changes modern-target LZMA2 async decoding, not raw LZMA or legacy-target decoding. Each compressed LZMA2 chunk declares at most 64 KiB of compressed data. That payload is filled asynchronously into the range decoder's existing pooled fast-input lease. The existing optimized decoder then performs CPU-only work on that buffer; it never makes a synchronous call to the supplied input stream. There is no second request-sized compressed-input allocation. CPU-only batches are capped at **16 KiB of output between cancellation checks**, even when the caller asks for much more. The fast probability bank is reused across continuation chunks and reinitialized on property/state resets. Scalar models are not also allocated for that path. The decoder still validates LZMA2 compressed/uncompressed counts, end markers, pending output, and outer XZ checks. Input refills are bounded by the declared payload, not the rented array's capacity. Incomplete fills cannot enter unsafe decoding; truncation retains the `DataErrorException` category, and disposal returns partially filled leases. Cancellation after a complete payload fill can resume without losing buffered data. Tests exercise yielding async-only non-seekable input, one-byte/short reads, following-chunk sentinels, malformed/truncated payloads, retries, oversized poisoned rents, cancellation, model reuse, and disposal. A fresh comparison against `31f3476` reuses the complete **32 MiB** mixed XZ workload and 64 KiB native-memory requests. Each version/input runs in a fresh process with two warmups and seven measured complete drains; version order is reversed for the file scenario. Release .NET 10 uses disabled tiered compilation. No builds or tests compete with these measurements. | Input / memory-async API | Before ms | After ms | Before process CPU ms | After process CPU ms | Before allocated bytes | After allocated bytes | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Memory | 5,963.0 | 769.2 | 5,961.2 | 769.2 | 80,672 | 72,600 | | File | 6,812.9 | 768.7 | 6,932.0 | 780.6 | 6,195,096 | 3,721,104 | This workload's elapsed medians improve by approximately **7.8x memory-backed** and **8.9x file-backed**; process CPU reductions support the decoding explanation, not merely reduced waiting. These results do not establish a speedup for raw LZMA, legacy targets, other data distributions, or cold storage. File allocations still include suspension costs and pool misses. All original checksum work remains. Artifacts and hash-checked before/after DLLs are in `fix3-harness/` in the parent directory. Its `evaluate.py`, `results.json`, `library-hashes.json`, and raw logs record the comparison. Both versions decode to the same byte count; the updated build additionally passes full SHA-256 verification of this workload through all three read APIs and all 24 byte-for-byte issue-sample combinations. All **969 targeted Release tests**, including 7z, pass. All six library targets and net48 test compilation pass; all 84 benchmark cases still smoke-test cleanly. The final complete **.NET 10 Release suite passes: 3,629 tests, no failures or skips**. A final measured-phase async memory trace confirms the intended path change: **81.4%** of workload leaf samples are in `CodeFastCore`, **8.2%** in CRC64, and **5.5%** in the fast matched-literal helper. Per-bit async decoder helpers no longer dominate. This trace contains 1,478 workload samples, one complete measured phase, and zero lost events. Its raw trace, Speedscope file, and analysis JSON are `fix3-harness/after-memory.*` / `after-memory-analysis.json`. Sampling limitations remain the same as in the original evaluation. The checksum tables remain shared and unchanged: the profiling report identified no per-block table-allocation problem to fix. ## Artifacts and reproduction The controlled comparison is isolated from preliminary sample-file measurements: ```text /private/var/folders/6c/j56r6kqn5yj7q1_c6vm5c0t80000gn/T/opencode/xz-bcj-profile/evaluate/ ``` This directory contains `prepare.py`, `run.py`, `trace.py`, the shared harness source, separate baseline/pooled source exports and harness builds, library hashes, the evaluated library diff, repository state, fixture generator and hashes, raw timing logs, JSON results, traces, Speedscope files, and analyzer source. Preparation and runs do not change the repository checkout. ```sh python3 prepare.py python3 run.py python3 trace.py ``` `prepare.py` anchors its source comparison to `0aaf9b92`, builds both libraries and harnesses, and creates the fixture. `run.py` performs the sequential untraced comparison. `trace.py` separately rebuilds the harness/analyzer before collecting traces; do not run it concurrently with measurements. Open `.speedscope.json` files in . Filenames identify version, filter/API, input source, and read size. Raw Speedscope files include warmup; the analysis JSON uses only the measured phase. `XzBcjBenchmarks` provides repeatable repository benchmarks with built-in inputs: ```sh dotnet run --project tests/SharpCompress.Performance -c Release -- --filter '*XzBcjBenchmarks*' ``` These BenchmarkDotNet cases are not the controlled before/after harness above. They cover all six isolated filters and a built-in 1 MiB x86+LZMA2 XZ fixture, with 1-byte, 16-byte, 10,240-byte, and 1 MiB requests. Set `SHARPCOMPRESS_XZ_PROFILE_FILE` to substitute a local `.xz` file for the XZ cases; isolated-filter cases are unchanged. ## Validation - The complete **.NET 10 Release suite passes: 3,571 tests, no failures or skips**. - All **718 targeted XZ, branch-converter, LZMA, and Tar tests** pass on .NET 10. - New lifetime tests verify oversized poisoned rents, exact-once return, reads after disposal, cancellation/read failures, early disposal, block completion, corrupt checksums, and caller-owned input preservation. Repeated filter construction has a bounded-allocation regression test. - The issue #1471 `7z2605-linux-x64.tar.xz` sample matches liblzma byte-for-byte in all **24** combinations of sync/async overload, request size, and destination offset. All 6,958,080 decoded bytes and the XZ checksum match. - Both isolated comparison builds decode the complete synthetic workload to the expected SHA-256 through sync and both async read overloads. - All **84 repository benchmark cases** were smoke-tested for output lengths and XZ checks. These smoke runs are not the before/after timing measurements. - All six library target frameworks build without warnings; net48 tests compile. net48 tests were not executed on macOS.