sharpcompress/docs/XZ_PROFILING.md

# XZ BCJ pooling and decoder profiling

## Branch and changes

Evaluated on 2026-10-24 on `fix/xz-bcj-streaming`. The baseline is
`0aaf9b92` (the streaming BCJ correctness fix); the pooled version is
`5c46363f`. Both libraries are clean Release builds in separate source exports,
using the same harness and inputs. The original broken BCJ implementation is
not a meaningful performance baseline: it can omit conversions or corrupt
output, so it would not perform equivalent work.

Changes evaluated:

- The shared 8 KiB BCJ buffer is rented from `ArrayPool<byte>.Shared` and returned
  exactly once on disposal. Processing is bounded by the logical buffer size and
  valid input count, not an oversized rent's capacity.
- All six XZ executable filters use that buffer: x86, ARM, ARM-Thumb, PowerPC,
  SPARC, and IA-64. Unprocessed instruction tails remain buffered between reads.
- XZ blocks retain their filter objects for cleanup. Advancing to another block
  disposes the previous block, and disposing an XZ stream releases its current
  block. This returns the BCJ buffer and the LZMA2 decoder's existing pooled
  dictionary/range buffers; previously the filter chain was not disposed.
- Filters dispose their own wrappers, not caller-owned input streams. The LZMA2
  wrapper explicitly leaves its compressed input open. Disposed readers reject
  further reads before they can use or rent buffers again.

No range-coder, LZMA decoding, checksum, or async scheduling optimization is
included in the pooled comparison.

The saved library diff was checked against these commits byte-for-byte. The
exported baseline DLL SHA-256 is
`f7c9c5ac700315205268a0f1aec0c8517289e6f81bfd9d1459d0ec3c17d5985c`;
the pooled DLL SHA-256 is
`ab7cee7f00bcc5e37f967ca0f74ff3c34a8eb93ee128cf08ca41488987106961`.

## Workloads and measurement

### Complete XZ decoding

The controlled workload is **16.36 MiB compressed** (17,159,680 bytes),
**32 MiB decoded** (33,554,432 bytes), in one XZ block using x86 BCJ, LZMA2,
an 8 MiB dictionary, and CRC64. Python 3.9.6's liblzma-backed `lzma.compress`
created it with preset 6. A generator seeded with `1471` alternates 64 KiB of
random bytes and 64 KiB of structured x86 records. Each 16-byte structured
record contains CALL and JMP instructions targeting fixed absolute addresses.
This exercises both incompressible data and BCJ-friendly instructions.

Compressed SHA-256:
`d1b182144055a3d1767944f32081bc2142f703372dae9d28f5073b629a733e1b`.

Decoded SHA-256:
`9150fb1e85094a06b7aeb197908a30d3a8466e4d298f629d13b3f0ca99243715`.

Every extraction drains through the XZ checksum and verifies the decoded byte
count. Memory-backed scenarios preload compressed bytes outside measurement.
File-backed scenarios use a 4 KiB `FileStream` buffer, `SequentialScan`, and
`FileOptions.Asynchronous`. Filesystem-cache effects are warm, not cold-storage
performance. The synchronous API, async array API (`Task<int>`), and async memory
API (`ValueTask<int>`) all use 64 KiB requests with a nonzero destination offset.

Each version/scenario runs in a fresh process with two warmups and seven measured
extractions. Version order alternates between scenarios. Release `net10.0` runs
disable tiered compilation. No builds or tests compete with the timed runs.
Measurements use .NET 10.0.13 on Apple M4/macOS 27.0.1 ARM64.

Allocation figures are exact process-wide `GC.GetTotalAllocatedBytes(true)`
deltas around extraction **and disposal**. They include the caller buffer and
stream construction, but exclude compressed-input preloading, process startup,
and between-iteration logging. They are allocation counts, not live memory or
peak working set. The pooled decoder still needs an 8 MiB dictionary and an
8 KiB BCJ buffer while decoding.

### Isolated executable filters

Each of the six filters separately decodes 1 MiB of repeated architecture-specific
encoded instructions, without LZMA2 or XZ checksum work. Request sizes are 1, 16,
and 10,240 bytes, using sync and both async read overloads. Input setup is excluded;
the destination buffer and filter lifetime are included. These artificial inputs
isolate wrapper/transform costs and do not represent typical executable files.

### Sampling traces

Separate traces use `dotnet-trace` 10.0.750501 with `dotnet-sampled-thread-time`, CLR GC
allocation ticks, and custom measured-phase markers. XZ traces contain three
extractions after two warmups. Each isolated-filter trace contains 10,000
iterations after warmup. TraceEvent 3.1.21 performs analysis; startup and warmup
events are excluded using the phase markers.

Stack shares are sampled managed thread time, not exact CPU instruction counts.
The report separates stacks containing SharpCompress work from unrelated runtime
background stacks. File-backed stacks can include waits/safepoints, and inlining
can attribute helper costs to their caller. Allocation ticks provide sampled site
attribution; the untraced exact allocation deltas are the authoritative totals.

## Fresh results

Medians per complete **32 MiB** XZ extraction, including construction and disposal:

| Input / API | Baseline ms | Pooled ms | Baseline allocated bytes | Pooled allocated bytes |
| --- | ---: | ---: | ---: | ---: |
| Memory / sync | 943.8 | 869.3 | 8,482,480 | 69,584 |
| Memory / async array | 6,806.8 | 6,718.0 | 8,952,504 | 523,184 |
| Memory / async memory | 6,607.1 | 6,770.7 | 8,952,504 | 523,184 |
| File / sync | 1,128.0 | 898.2 | 8,486,704 | 74,096 |
| File / async array | 7,892.0 | 7,885.2 | 15,116,192 | 8,695,936 |
| File / async memory | 7,899.9 | 17,081.7 | 14,392,056 | 10,287,064 |

The memory-backed comparison saves approximately **8 MiB per extraction**:
**99.2% of warm sync allocations** and **94.2% of warm async allocations**.
Most of this comes from returning the existing pooled LZMA2 dictionary; the BCJ
buffer alone accounts for only about 8 KiB. Pooling the BCJ buffer without fixing
filter-chain disposal would not return either lease reliably.

This is **not evidence of a general throughput improvement**. Memory-backed
async process CPU medians were 6,508 ms before versus 6,520 ms after for the
array API, and 6,466 ms versus 6,518 ms for the memory API. The file/memory-async
case was substantially slower after the change in this run (process CPU medians
7,748 ms versus 10,064 ms, with additional elapsed waiting time). That result is
retained, not discarded. File-backed scheduling, suspension allocations, warm
pool behavior, and machine state can vary; the smaller file allocation reductions
should not be treated as precise attribution to a specific buffer. A separate
repeatability/async-I/O investigation is needed before claiming file throughput
benefits or explaining that slowdown.

### All six isolated filters

Synchronous 1 MiB drains with **10,240-byte reads**, including the destination
buffer and construction/disposal:

| Filter | Baseline ms | Pooled ms | Baseline allocated bytes | Pooled allocated bytes |
| --- | ---: | ---: | ---: | ---: |
| x86 | 0.387 | 0.388 | 19,064 | 10,528 |
| ARM | 0.614 | 0.607 | 18,992 | 11,128 |
| ARM-Thumb | 0.422 | 0.427 | 18,992 | 11,128 |
| PowerPC | 0.751 | 0.721 | 18,992 | 11,128 |
| SPARC | 0.763 | 0.817 | 18,992 | 11,128 |
| IA-64 | 0.325 | 0.328 | 18,992 | 11,128 |

The async memory API similarly saves about 8 KiB across all six architectures;
the non-x86 cases allocate roughly 19,064 bytes before versus 11,200 bytes after
(SPARC's baseline median was 18,776). Their pooled elapsed medians range from
0.327 ms for IA-64 to 0.798 ms for SPARC. The async array path still allocates
roughly 26 KiB per drain because its non-cached integer results require tasks.
Full per-architecture/per-overload results and all seven samples are in
`results.json`; these short microbenchmarks do not establish a CPU speedup.

### Small-read overhead remains

Isolated x86, **16-byte requests**, again per 1 MiB:

| API | Baseline allocated bytes | Pooled allocated bytes | Baseline ms | Pooled ms |
| --- | ---: | ---: | ---: | ---: |
| Sync | 8,504 | 304 | 0.660 | 0.743 |
| Async array | 9,446,240 | 9,438,376 | 1.709 | 1.796 |
| Async memory | 9,056 | 1,192 | 3.437 | 3.436 |

The array API's approximately 9 MiB remains after pooling: each 16-byte result
is outside the runtime's small integer task cache and passes through multiple
`Task<int>` wrappers. By contrast, 1-byte results can use cached tasks. The memory
path has low allocations but substantial call/state-machine overhead for tiny
reads; low allocations do not imply it is faster than the array path.

### CPU and sampled allocation attribution

All **18 traces** contain one complete measured phase and report **zero lost
events**. The pooled memory-backed sync trace attributes 82.6% of workload leaf
samples to `Decoder.CodeFastCore`, 8.8% to CRC64, 4.4% to the fast matched-literal
helper, and 2.1% to x86 BCJ conversion. The corresponding baseline shares are
83.2%, 8.4%, 4.8%, and 2.0%: pooling does not move the main CPU bottleneck.

The pooled memory-backed async trace instead attributes 30.6% of workload leaf
samples to literal `DecodeNormalAsync`, 16.7% to `BitDecoder.DecodeAsync`, 12.6%
to async builder `Start`, and 11.3% to `Normalize2Async`. These per-symbol async
operations dominate despite the input being in memory. The corresponding
baseline distribution is similar. There is no optimized `CodeFastCore` hotspot
on this path.

The isolated pooled-filter traces attribute 88.4–94.5% of workload leaf samples
to the architecture's converter (x86 91.5%, ARM 93.6%, ARM-Thumb 92.4%, PowerPC
94.4%, SPARC 94.5%, IA-64 88.4%). Buffer copying accounts for roughly 4–8%.
These are instruction-heavy synthetic inputs; their rankings are not a
cross-architecture throughput prediction for real executables.

Baseline memory-backed XZ allocation ticks attribute approximately 24 MiB across
three measured extractions to dictionary arrays at `OutWindow.EnsureCapacity`.
The pooled memory-backed measured phases contain no sampled allocations at that
site; remaining async samples principally identify `Task<int>` results in the
read chain. All six pooled isolated-filter traces similarly contain no sampled
recurring BCJ-buffer allocations at `SetBaseStream`; their output buffers are
still allocated by the harness. Per-site sampling weights are approximate and
must not be substituted for the exact allocation table above.

File-backed traces still contain dictionary pool misses and suspended async
state-machine allocations. Most file-backed thread-time leaf samples are in
`Monitor.Enter_Slowpath`; these sampled stacks include waits/safepoints and do
**not** demonstrate that a lock consumes 98% of instruction CPU time. They do
not explain the adverse elapsed result by themselves, nor justify a lock rewrite.

## Remaining opportunities

These describe the profiled pooling baseline; completed follow-ups are recorded
separately below rather than rewriting the historical measurements.

- Async LZMA2 currently calls the per-symbol async decoder, whereas sync decoding
  can use the optimized range-decoder path. A future experiment should separate
  asynchronous input refills from CPU-only decoding. It must not replace async I/O
  with blocking reads, weaken cancellation, overread a chunk, or bypass corruption
  checks. This evaluation does not make that change.
- Small async array reads can allocate a `Task<int>` at multiple wrapper layers
  for a non-cached read result. Pooling an 8 KiB buffer cannot remove that cost.
  Pure delegation and a ValueTask-based internal read chain are separate candidates,
  with exception/cancellation semantics requiring regression coverage.
- Calling the memory overload on `XZStream` does not make its whole chain
  allocation-free: the stream still uses its array-based async implementation.
  The isolated BCJ memory path and complete XZ memory path must be measured
  separately rather than assuming the overload name guarantees equivalent costs.
- Checksum lookup tables are shared, not per-block scratch allocations. There is
  no evidence here to justify pooling them. Checksum work must remain enabled in
  future throughput comparisons.

## Follow-up: remove duplicate BCJ result tasks

On `perf/xz-async-decoding`, the first follow-up keeps the internal BCJ read as
a `ValueTask<int>` and creates a task only at the existing public array boundary.
Already-buffered reads complete without entering an async state machine. Custom
subclasses installing a synchronously throwing input stream retain the previous
faulted/canceled-task behavior; sync signatures are unchanged.

A fresh before/after comparison against the pooled baseline uses the same isolated
1 MiB x86 workload, 16-byte array reads, two warmups, seven measured drains,
Release .NET 10, and disabled tiered compilation. Median allocations fall from
**9,438,376 to 4,719,784 bytes**; median elapsed time is **1.695 to 0.963 ms**.
This is a small-read wrapper result, not a complete XZ throughput claim. Logs and
the follow-up harness are in the parent artifact directory as `fix1-before.log`,
`fix1-after.log`, and `fix1-harness/`.

All **728 targeted Release tests** pass, including allocation guards for all six
architectures and fault/cancellation regressions. All six library targets build,
and net48 test compilation passes. The remaining public `Task<int>` result cannot
be removed without changing that API; a native memory-based XZ chain remains a
separate follow-up.

## Follow-up: native XZ memory read chain

The next change adds native memory/ValueTask reads to `XZStream` and `XZBlock`
on modern targets, carries memory-based I/O through BCJ refills, and completes
already-buffered reads without traversing multiple async wrappers. All buffered
bytes still update the block check exactly once. Array API signatures and sync
array-based I/O remain unchanged. Native/custom destination memory works with
all four XZ check types; SHA-256 uses bounded pooled scratch when required.

A fresh comparison against the first follow-up (`de7a5d9`) uses the built-in
1 MiB x86+LZMA2 fixture, 16-byte memory reads at offset 5, two warmups, seven
measured complete drains, Release .NET 10, and disabled tiered compilation.
Median allocations fall from **14,180,048 to 15,056 bytes**, with elapsed medians
**22.399 to 19.422 ms**. Both builds retain checksum verification and return the
expected output length. This does not yet address the async range-decoder CPU
bottleneck or establish file-backed throughput improvements.

Artifacts are `fix2-before.log`, `fix2-after.log`, and `fix2-harness/` in the parent
directory. The harness's before/after DLLs were explicitly bound and hash-checked
after observing that MSBuild reused an earlier reference when switching HintPath
properties; `fix2-harness/comparison.json` records the actual measured hashes.

All **764 targeted Release tests** pass, including yielding async-only input,
seekable/forward-only streams, native/custom memory, mixed async overloads,
bounded allocations, empty/canceled reads, and checksum corruption. All six
library targets and net48 test compilation pass. The issue sample still matches
liblzma in all 24 read-size/overload/offset combinations.

## Follow-up: bounded async LZMA2 fast decoding

The third follow-up changes modern-target LZMA2 async decoding, not raw LZMA or
legacy-target decoding. Each compressed LZMA2 chunk declares at most 64 KiB of
compressed data. That payload is filled asynchronously into the range decoder's
existing pooled fast-input lease. The existing optimized decoder then performs
CPU-only work on that buffer; it never makes a synchronous call to the supplied
input stream. There is no second request-sized compressed-input allocation.

CPU-only batches are capped at **16 KiB of output between cancellation checks**,
even when the caller asks for much more. The fast probability bank is reused
across continuation chunks and reinitialized on property/state resets. Scalar
models are not also allocated for that path. The decoder still validates LZMA2
compressed/uncompressed counts, end markers, pending output, and outer XZ checks.

Input refills are bounded by the declared payload, not the rented array's
capacity. Incomplete fills cannot enter unsafe decoding; truncation retains the
`DataErrorException` category, and disposal returns partially filled leases.
Cancellation after a complete payload fill can resume without losing buffered
data. Tests exercise yielding async-only non-seekable input, one-byte/short reads,
following-chunk sentinels, malformed/truncated payloads, retries, oversized
poisoned rents, cancellation, model reuse, and disposal.

A fresh comparison against `31f3476` reuses the complete **32 MiB** mixed XZ
workload and 64 KiB native-memory requests. Each version/input runs in a fresh
process with two warmups and seven measured complete drains; version order is
reversed for the file scenario. Release .NET 10 uses disabled tiered compilation.
No builds or tests compete with these measurements.

| Input / memory-async API | Before ms | After ms | Before process CPU ms | After process CPU ms | Before allocated bytes | After allocated bytes |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Memory | 5,963.0 | 769.2 | 5,961.2 | 769.2 | 80,672 | 72,600 |
| File | 6,812.9 | 768.7 | 6,932.0 | 780.6 | 6,195,096 | 3,721,104 |

This workload's elapsed medians improve by approximately **7.8x memory-backed**
and **8.9x file-backed**; process CPU reductions support the decoding explanation,
not merely reduced waiting. These results do not establish a speedup for raw
LZMA, legacy targets, other data distributions, or cold storage. File allocations
still include suspension costs and pool misses. All original checksum work remains.

Artifacts and hash-checked before/after DLLs are in `fix3-harness/` in the parent
directory. Its `evaluate.py`, `results.json`, `library-hashes.json`, and raw logs
record the comparison. Both versions decode to the same byte count; the updated
build additionally passes full SHA-256 verification of this workload through all
three read APIs and all 24 byte-for-byte issue-sample combinations.

All **969 targeted Release tests**, including 7z, pass. All six library targets
and net48 test compilation pass; all 84 benchmark cases still smoke-test cleanly.
The final complete **.NET 10 Release suite passes: 3,629 tests, no failures or skips**.

A final measured-phase async memory trace confirms the intended path change:
**81.4%** of workload leaf samples are in `CodeFastCore`, **8.2%** in CRC64,
and **5.5%** in the fast matched-literal helper. Per-bit async decoder helpers no
longer dominate. This trace contains 1,478 workload samples, one complete measured
phase, and zero lost events. Its raw trace, Speedscope file, and analysis JSON are
`fix3-harness/after-memory.*` / `after-memory-analysis.json`. Sampling limitations
remain the same as in the original evaluation.

The checksum tables remain shared and unchanged: the profiling report identified
no per-block table-allocation problem to fix.

## Artifacts and reproduction

The controlled comparison is isolated from preliminary sample-file measurements:

```text
/private/var/folders/6c/j56r6kqn5yj7q1_c6vm5c0t80000gn/T/opencode/xz-bcj-profile/evaluate/
```

This directory contains `prepare.py`, `run.py`, `trace.py`, the shared harness
source, separate baseline/pooled source exports and harness builds, library
hashes, the evaluated library diff, repository state, fixture generator and
hashes, raw timing logs, JSON results, traces, Speedscope files, and analyzer
source. Preparation and runs do not change the repository checkout.

```sh
python3 prepare.py
python3 run.py
python3 trace.py
```

`prepare.py` anchors its source comparison to `0aaf9b92`, builds both libraries
and harnesses, and creates the fixture. `run.py` performs the sequential untraced
comparison. `trace.py` separately rebuilds the harness/analyzer before collecting
traces; do not run it concurrently with measurements. Open `.speedscope.json`
files in <https://www.speedscope.app/>. Filenames identify version, filter/API,
input source, and read size. Raw Speedscope files include warmup; the analysis JSON
uses only the measured phase.

`XzBcjBenchmarks` provides repeatable repository benchmarks with built-in inputs:

```sh
dotnet run --project tests/SharpCompress.Performance -c Release -- --filter '*XzBcjBenchmarks*'
```

These BenchmarkDotNet cases are not the controlled before/after harness above.
They cover all six isolated filters and a built-in 1 MiB x86+LZMA2 XZ fixture, with
1-byte, 16-byte, 10,240-byte, and 1 MiB requests. Set
`SHARPCOMPRESS_XZ_PROFILE_FILE` to substitute a local `.xz` file for the XZ cases;
isolated-filter cases are unchanged.

## Validation

- The complete **.NET 10 Release suite passes: 3,571 tests, no failures or skips**.
- All **718 targeted XZ, branch-converter, LZMA, and Tar tests** pass on .NET 10.
- New lifetime tests verify oversized poisoned rents, exact-once return, reads
  after disposal, cancellation/read failures, early disposal, block completion,
  corrupt checksums, and caller-owned input preservation. Repeated filter
  construction has a bounded-allocation regression test.
- The issue #1471 `7z2605-linux-x64.tar.xz` sample matches liblzma byte-for-byte in
  all **24** combinations of sync/async overload, request size, and destination
  offset. All 6,958,080 decoded bytes and the XZ checksum match.
- Both isolated comparison builds decode the complete synthetic workload to the
  expected SHA-256 through sync and both async read overloads.
- All **84 repository benchmark cases** were smoke-tested for output lengths and
  XZ checks. These smoke runs are not the before/after timing measurements.
- All six library target frameworks build without warnings; net48 tests compile.
  net48 tests were not executed on macOS.