From c7a72bce13c18e5a6929f0414b34bc6c7ceb02e7 Mon Sep 17 00:00:00 2001 From: Natalia Portillo Date: Sun, 12 Apr 2026 13:23:39 +0100 Subject: [PATCH] Update specification. --- docs/spec/blocks/erasure_coding.adoc | 273 +++++++++++++++++++++++++++ 1 file changed, 273 insertions(+) diff --git a/docs/spec/blocks/erasure_coding.adoc b/docs/spec/blocks/erasure_coding.adoc index f16f290..8334907 100644 --- a/docs/spec/blocks/erasure_coding.adoc +++ b/docs/spec/blocks/erasure_coding.adoc @@ -1,5 +1,278 @@ === 🛡️ Erasure Coding Map Block (`ECMB`) +This block stores the master recovery map for erasure-coded images. +It enables reconstruction of corrupted data blocks, DDT tables, metadata blocks, and the index using Reed-Solomon parity computed over the raw on-disk bytes (block header + compressed payload). + +The ECMB is *not* referenced by the index, because the index itself may need recovery. +Instead, it is located via the <> at the end of the file. +A duplicate copy of the ECMB is written immediately after the primary copy for redundancy. + +==== Feature Flag + +Erasure coding uses the `featureCompatibleRo` field in the image header: + +[cols="2,2,6",options="header"] +|=== +|Bit |Name |Description +|0 |`AARU_FEATURE_ROCOMPAT_ERASURE` |Image contains erasure coding parity blocks, ECMB, and recovery footer. Readers that do not understand this bit SHOULD open the image read-only but CAN safely ignore the parity data. +|=== + +Older readers that do not understand this bit will skip the parity blocks and ECMB, and read the image normally. + +==== Design Rationale + +Parity is computed on the *raw on-disk bytes* (block header + compressed payload), not on uncompressed sector data. +This is because a single bit flip in a compressed LZMA, Zstandard, or FLAC block causes decompression to fail entirely—uncompressed parity would be useless for the exact scenario it is supposed to fix. +Recovery reconstructs the original on-disk bytes, then decompresses normally. + +Parity blocks are always written *uncompressed*. +Parity of compressed data is pseudo-random and incompressible—attempting compression wastes CPU time for zero benefit. + +==== Structure Definition + +[source,c] +---- +#define ECMB_MAGIC 0x424D4345 // "ECMB" in little-endian ASCII + +typedef struct ErasureCodingMapHeader +{ + uint32_t identifier; // Block identifier, must be 0x424D4345 + uint8_t algorithm; // Erasure coding algorithm + uint8_t stripeGroupCount; // Number of stripe groups in payload + uint16_t compression; // Compression algorithm for the mapping payload + uint64_t cmpLength; // Size in bytes of the compressed mapping payload + uint64_t length; // Size in bytes of the uncompressed mapping payload + uint64_t cmpCrc64; // CRC64-ECMA of the compressed mapping payload + uint64_t crc64; // CRC64-ECMA of the uncompressed mapping payload +} ErasureCodingMapHeader; +---- + +==== Field Descriptions + +[cols="2,2,2,6",options="header"] +|=== +|Type |Size |Name |Description +|uint32_t |4 bytes |identifier |Block identifier, always `ECMB` (0x424D4345). +|uint8_t |1 byte |algorithm |Erasure coding algorithm (see table below). +|uint8_t |1 byte |stripeGroupCount |Number of stripe groups in the payload (typically 1–5). +|uint16_t |2 bytes |compression |Compression algorithm used for the ECMB payload (same enumeration as data blocks). +|uint64_t |8 bytes |cmpLength |Size of the compressed ECMB payload in bytes. +|uint64_t |8 bytes |length |Size of the uncompressed ECMB payload in bytes. +|uint64_t |8 bytes |cmpCrc64 |CRC64-ECMA of the compressed ECMB payload. +|uint64_t |8 bytes |crc64 |CRC64-ECMA of the uncompressed ECMB payload. +|=== + +==== Erasure Coding Algorithms + +[cols="1,2,6",options="header"] +|=== +|Value |Name |Description +|0 |XOR |Simple XOR parity. M must be 1. Fastest but tolerates only a single block loss per stripe. +|1 |RS-Vandermonde |Reed-Solomon with Vandermonde generator matrix over GF(2^8). Supports M ≥ 1. Tolerates up to M block losses per stripe. See the Reed-Solomon appendix for details. +|=== + +==== Payload Structure + +The payload (after decompression) contains `stripeGroupCount` consecutive stripe groups. +Each group starts with a `StripeGroupDescriptor`, followed by its stripe descriptors. + +===== Stripe Group Descriptor + +[source,c] +---- +typedef struct StripeGroupDescriptor +{ + uint8_t groupType; // Protection group type + uint16_t K; // Number of data blocks per stripe + uint16_t M; // Number of parity blocks per stripe + uint32_t shardSize; // Shard size used for RS math (see below) + uint32_t stripeCount; // Number of stripes in this group + uint16_t interleaveDepth; // Interleave depth D (1 = consecutive) +} StripeGroupDescriptor; +---- + +[cols="2,2,2,6",options="header"] +|=== +|Type |Size |Name |Description +|uint8_t |1 byte |groupType |Protection group type (see table below). +|uint16_t |2 bytes |K |Number of data blocks per stripe. +|uint16_t |2 bytes |M |Number of parity blocks per stripe. +|uint32_t |4 bytes |shardSize |Shard size in bytes for RS math. Parity buffers are allocated at this size. Actual blocks that are smaller are zero-padded to this size for RS computation. +|uint32_t |4 bytes |stripeCount |Number of stripes in this group. +|uint16_t |2 bytes |interleaveDepth |Interleave depth D. D=1 for consecutive assignment (current implementation). Reserved for future interleaving support. +|=== + +===== Protection Group Types + +[cols="1,2,6",options="header"] +|=== +|Value |Name |Description +|0 |Data |User data blocks (DBLK with DataType=UserData). Consecutive stripe assignment. +|1 |DDT-Secondary |Secondary DDT subtables. Batch parity at finalization. +|2 |DDT-Primary |Primary DDT block. Batch parity (K=1 typical, producing M replicas). +|3 |Metadata |All non-data, non-DDT, non-index blocks: media tags (DBLK with non-UserData types), TracksBlock, ChecksumBlock, GeometryBlock, MetadataBlock, DumpHardwareBlock, CicmBlock, AaruMetadataJsonBlock, etc. Single batch stripe at finalization. +|4 |Index |Index block. Batch parity (K=1, producing M replicas)—strongest protection for this critical structure. +|=== + +===== Stripe Descriptor + +Each stripe contains `actualK` data block entries followed by `M` parity block entries: + +[source,c] +---- +// Per stripe: +uint16_t actualK; // Actual data blocks in this stripe (≤ K; last stripe may be partial) + +// Repeated actualK times: +typedef struct StripeDataBlockEntry +{ + uint64_t offset; // Absolute file offset of the data block + uint32_t onDiskSize; // Actual on-disk bytes (header + compressed payload) + uint64_t shardCrc64; // CRC64-ECMA of on-disk bytes (onDiskSize bytes, NOT zero-padded) +} StripeDataBlockEntry; + +// Repeated M times: +typedef struct StripeParityBlockEntry +{ + uint64_t offset; // Absolute file offset of the parity DBLK +} StripeParityBlockEntry; +---- + +The `shardCrc64` is computed over exactly `onDiskSize` bytes of the block's raw on-disk data. +This is independent of `BlockHeader.cmpCrc64`—it can detect corruption even when the `BlockHeader` itself is garbled. + +==== Parity Block Storage + +Parity shards are stored as standard data blocks (`DBLK`) with the following `DataType` values: + +[cols="1,2,6",options="header"] +|=== +|Value |Name |Description +|104 |ErasureParity |Parity for user data blocks. +|105 |ErasureParityDdt |Parity for DDT secondary blocks. +|106 |ErasureParityDdtPrimary |Parity for DDT primary block. +|107 |ErasureParityMeta |Parity for metadata blocks. +|108 |ErasureParityIndex |Parity for the index block. +|=== + +Parity blocks are always written *uncompressed* (`compression = kCompressionNone`). +The parity `length` and `cmpLength` fields equal the maximum actual on-disk block size across all blocks in the stripe—only the non-zero prefix of each shard participates in the RS computation (since `GF_mul(0, c) = 0` for any coefficient `c`). +This means parity size scales with the *actual compressed* block size, not with the theoretical maximum uncompressed block size. + +==== Stripe Assignment (Data Group) + +Data blocks are assigned to stripes consecutively: + +---- +Blocks 0..K-1 → Stripe 0 +Blocks K..2K-1 → Stripe 1 +Blocks 2K..3K-1 → Stripe 2 +... +Remaining blocks → Last stripe (partial, actualK < K) +---- + +This produces at most one partial stripe at the end, giving correct M/K overhead regardless of image size. + +NOTE: Interleaved (round-robin) assignment (`block i → slot i mod K`) was considered but rejected because it produces K partial stripes for images with fewer than K² blocks, resulting in overhead far exceeding the intended M/K ratio. + +==== Shard Size and Parity Sizing + +Parity computation uses the following optimizations: + +* Each block in a stripe contributes only its actual on-disk bytes (`onDiskSize`) to parity accumulation—the remaining bytes to `shardSize` are implicitly zero. +* The parity block `length`/`cmpLength` is set to `max(onDiskSize)` across all blocks in the stripe, not to `shardSize`. Parity size scales with actual compressed block sizes. +* Parity buffer allocation is *deferred* until the first data block is written (lazy allocation), using the first block's actual size plus 25% headroom. Buffers grow via `realloc` if a later block is larger. This avoids pre-allocating gigabytes based on theoretical maximum block sizes. + +==== Metadata Group Block Size Determination + +The metadata group protects blocks with varying header formats: + +* **DataBlock** entries (media tags, sector prefix/suffix, subchannel, etc.): on-disk size computed as `sizeof(BlockHeader) + cmpLength`, read directly from the block's `BlockHeader`. +* **Non-DataBlock** entries (TracksBlock, ChecksumBlock, GeometryBlock, DumpHardwareBlock, MetadataBlock, CicmBlock, AaruMetadataJsonBlock): these block types have different header layouts with no `cmpLength` field at the `BlockHeader` offset. On-disk size is computed from the gap between consecutive index entry offsets. + +CAUTION: Reading a non-DataBlock header as a `BlockHeader` and accessing the `cmpLength` field produces garbage values, as the bytes at that offset correspond to entirely different fields in each block type's header struct. + +==== Recovery Footer + +The last 160 bytes of the file contain the recovery footer, enabling the ECMB to be found even when the header or index is destroyed: + +[source,c] +---- +typedef struct AaruRecoveryFooter +{ + uint64_t ecmbOffset; // Absolute file offset of the primary ECMB + uint64_t ecmbLength; // Total on-disk size of the ECMB (header + payload) + uint64_t headerCrc64; // CRC64-ECMA of the original AaruHeaderV2 + AaruHeaderV2 backupHeader; // Complete copy of AaruHeaderV2 (128 bytes) + uint64_t footerMagic; // Must be 0x52464D4345525641 ("AVRECMFR") +} AaruRecoveryFooter; +---- + +[cols="2,2,2,6",options="header"] +|=== +|Type |Size |Name |Description +|uint64_t |8 bytes |ecmbOffset |Absolute file offset of the primary ECMB. +|uint64_t |8 bytes |ecmbLength |Total on-disk size of the ECMB (header + payload). +|uint64_t |8 bytes |headerCrc64 |CRC64-ECMA of the original `AaruHeaderV2` at file offset 0. +|AaruHeaderV2 |128 bytes |backupHeader |Complete backup copy of the file header. +|uint64_t |8 bytes |footerMagic |Must be `0x52464D4345525641` ("AVRECMFR" in ASCII little-endian). +|=== + +A duplicate ECMB is written at `ecmbOffset + ecmbLength` (aligned) for additional redundancy. + +==== Recovery Chain + +The recovery chain defines the fallback sequence when structures are damaged: + +[cols="3,7",options="header"] +|=== +|Scenario |Recovery Path +|Normal operation |Header → `indexOffset` → Index → ECMB (via footer) → verify/recover +|Header corrupt |Footer `backupHeader` (verified by `headerCrc64`) → same as normal +|Index corrupt |Footer `ecmbOffset` → ECMB → index parity stripe → RS-reconstruct index +|DDT corrupt |ECMB → DDT parity stripe → RS-reconstruct DDT block +|Data block corrupt |ECMB → data parity stripe → RS-reconstruct on-disk bytes → decompress +|Metadata corrupt |ECMB → metadata parity stripe → RS-reconstruct +|ECMB corrupt |Duplicate ECMB at secondary offset +|=== + +==== File Layout + +The erasure coding structures are written at the end of the file: + +---- +[AaruHeaderV2] offset 0, 128 bytes +[Data blocks with consecutive parity] variable size +[DDT blocks (all levels)] variable size +[DDT parity blocks] M blocks (uncompressed) +[Metadata blocks] variable size +[Metadata parity blocks] M blocks (uncompressed) +[Index block] at header.indexOffset +[Index parity blocks] M blocks (uncompressed) +[ECMB (primary)] at footer.ecmbOffset +[ECMB (duplicate)] at ecmbOffset + ecmbLength (aligned) +[Recovery Footer] last 160 bytes of file +---- + +==== Automatic K,M Selection + +The library provides `aaruf_set_erasure_coding_auto(context, recovery_percent)` which computes K and M from a target recovery percentage (1–100): + +[cols="1,1,1,4",options="header"] +|=== +|Percent |M |K |Notes +|1% |2 |200 |Minimum M=2 for burst tolerance +|5% |2 |40 | +|10% |2 |20 | +|15% |3 |20 |M starts growing with higher percentages +|25% |5 |20 |Strong burst tolerance +|50% |8 |16 |M capped at 8 +|100% |8 |8 |Maximum protection +|=== + +Higher recovery percentages produce higher M values (more parity blocks per stripe = better burst-corruption tolerance). K stays roughly constant (16–40) to maintain reasonable stripe sizes and memory usage. M is computed as `clamp(round(20 × percent / 100), 2, 8)`, then `K = M × 100 / percent`. +=== 🛡️ Erasure Coding Map Block (`ECMB`) + This block stores the master recovery map for erasure-coded images. It enables reconstruction of corrupted data blocks, DDT tables, metadata blocks, and the index using Reed-Solomon or XOR parity computed over the raw on-disk bytes (block header + compressed payload).