aaru-dps/libaaruformat
1
0
Fork 0
mirror of https://github.com/aaru-dps/libaaruformat.git synced 2025-12-16 19:24:40 +00:00
Code Issues Packages Projects Releases Wiki Activity

[Specification] Add deduplication table version 2 (DDT2) definition

Browse Source
This commit is contained in:
Natalia Portillo
2025-07-31 16:07:38 +01:00
parent f87d75676a
commit 36ecda2f9a
2 changed files with 229 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/spec/spec.adoc
View File

@@ -51,3 +51,7 @@ include::structs/data.adoc[]
<<<
include::structs/ddt.adoc[]
<<<
include::structs/ddt2.adoc[]

224
docs/spec/structs/ddt2.adoc Normal file
View File

@@ -0,0 +1,224 @@
=== The deduplication table (`DDT2`)
The deduplication table is a multi-level table of pointers to LBAs contained in the image.
It starts with the following header.
[source,c]
/* Undefined */
==== Field Descriptions
[cols="2,2,2,6",options="header"]
|===
|Type
|Size
|Name
|Description
|uint32_t
|4 bytes
|identifier
|The deduplication table identifier, always `DDT2` or `DDTS`. The first level of a table is always `DDT2` and its presence is mandatory. Subtables will have `DDTS`
|uint16_t
|2 bytes
|type
|The data type pointed by this table. See Annex B.
|uint16_t
|2 bytes
|compression
|The compression algorithm used in the table. See Annex C.
|uint8_t
|1 byte
|levels
|How many levels of subtables are present. 1 means this is the only level.
|uint8_t
|1 byte
|tableLevel
|What level does this table correspond to
|uint64_t
|8 bytes
|previousLevel
|Pointer to absolute byte offset in the image file where the previous table level resides
|uint16_t
|2 bytes
|negative
|The negative displacement of LBA numbers. For media that can have negative LBAs, this establishes the number to substract to the table entry number
|uint64_t
|8 bytes
|start
|The first LBA contained in this table. It must be 0 for ‘DDT2’ blocks and can be other number for subtables ‘DDTS’
|uint8_t
|1 byte
|alignment
|Shift of alignment of all blocks in the image. This must be the same in all deduplication tables and subtables.
|uint8_t
|1 byte
|shift
|The shift used to calculate the position of a sector in a data block pointed by this table, or how many sectors are pointed by the next level.
|uint8_t
|1 byte
|sizeType
|Size type (see table below)
|uint64_t
|8 bytes
|entries
|How many pointers follow this header.
|uint32_t
|4 bytes
|cmpLength
|The size in bytes of the compressed table that follows this header.
|uint32_t
|4 bytes
|length
|The size in bytes of the table block when decompressed.
|uint64_t
|8 bytes
|cmpCrc64
|The CRC64-ECMA checksum of the compressed table that follows this header.
|uint64_t
|8 bytes
|crc64
|The CRC64-ECMA checksum of the decompressed table.
|===
The size type defines the following type of entries:
[cols="1,1,6",options="header"]
|===
|Type
|Value
|Description
|Mini
|0
|Each entry uses two bytes, with the leftmost byte (mask 0xFF00) used for flags, and the rightmost byte used as a pointer to the sector or next level.
|Small
|1
|Each entry uses three bytes, with the leftmost byte used for flags and the next two bytes used as a pointer to the sector or next level.
|Medium
|2
|Each entry uses four bytes, with the leftmost byte (mask 0xFF000000) used for flags and the next three bytes used as a pointer to the sector or next level.
|Big
|3
|Each entry uses five bytes, with the leftmost byte used for flags and the next three bytes used as a pointer to the sector or next level.
|===
==== Sector Pointer Resolution and Table Levels
When `levels` is equal to 1—indicating a single-level deduplication table—each entry in the table corresponds directly to a media sector.
The pointer value is resolved using the following procedure:
- Right-shift the raw pointer value by the `shift` value.
- Multiply the result by the `alignment` to compute the absolute byte offset of the target data block.
- The remainder of the original pointer value modulo `(1 << shift)` yields the item index within the block.
Each data block stores a fixed number of bytes per sector, allowing compact and efficient sector addressing.
_For example_:
Given a pointer value of `0x8003`, a `shift` of 5, and an `alignment` of 9:
- `0x8003 >> 5 = 0x400 = 1024`
- `1024 * 9 = 9216`
- The sector index within the block is `0x8003 & 0x1F = 3`
Thus, the sector is located at byte offset `9216`, and it is the 3rd item in the block.
===== Multi-Level Tables
When `levels > 1`, the interpretation of pointer entries changes substantially.
Although typical usage involves no more than two levels, implementations **MUST** be capable of handling an arbitrary number of levels to ensure forward compatibility.
At each level—except the final—the table entry functions as an address to the next-level table.
The range of LBAs covered by each entry is calculated as:
[source]
range = entry_index * (1 << shift)^(levels - 1)
_For example_, with a `shift` value of 9 and two levels:
- Entry `0` spans LBAs `0–511`
- Entry `1` spans LBAs `512–1023`
With three levels:
- Entry `0` at level 0 spans LBAs `0–262143`
- Entry `0` at level 1 within that region spans LBAs `0–511`, and so on recursively.
===== Resolution Example
To locate sector `1012` using a two-level table with `shift = 9` and `alignment = 9`:
1. **Level 0**:
- Sector `1012` falls within entry `1` (covers `512–1023`)
- Entry `1` contains the value `0x12000`
- Multiply by `alignment` → `0x12000 * 9 = 0x225000 = 37,748,736`
- Read the next-level table at byte offset `37,748,736`, marked with the identifier `DDTS`
2. **Level 1**:
- The relevant entry is `500` (`1012 - 512 = 500`)
- Entry `500` contains `0x35006`
- Right-shift `0x35006 >> 9 = 0x6A = 106`
- Multiply by `alignment`: `106 * 9 = 954`
- Sector resides at byte offset `217,088` and is the 6th item in the block (`0x35006 & 0x1FF = 6`)
===== Deduplication table flags
[cols="2,1,6",options="header"]
|===
|Flag
|Value
|Description
|NotDumped
|`0x00`
|The sector(s) have not been dumped
|Dumped
|`0x01`
|The sector(s) have been dumped without errors
|Errored
|`0x02`
|The sector(s) returned an error on dumping
|Mode1Correct
|`0x03`
|The sector is MODE 1 and the suffix or prefix is correct and can be regenerated. Must only appear on deduplications tables with types CdSectorPrefixCorrected or CdSectorSuffixCorrected
|Mode2Form1Ok
|`0x04`
|The suffix for MODE 2 sectors is correct, can be regenerated, and corresponds to a MODE 2 Form 1 sector. Must only appear on deduplications tables with type CdSectorSuffixCorrected
|Mode2Form2Ok
|`0x05`
|The suffix for MODE 2 sectors is correct, can be regenerated, and corresponds to a MODE 2 Form 2 sector with a valid CRC. Must only appear on deduplications tables with type CdSectorSuffixCorrected
|Mode2Form2NoCrc
|`0x06`
|The suffix for MODE 2 sectors is correct, can be regenerated, and corresponds to a MODE 2 Form 2 sector with an empty CRC. Must only appear on deduplications tables with type CdSectorSuffixCorrected
|Twin
|`0x07`
|The pointer contains a “twin” sector table (see below)
|Unrecorded
|`0x08`
|The sector was unrecorded and each re-read returns random data
|===
When flags are present in a table that has sublevels it applies to all the sectors that shall be present in the subtable, unless the flag specify something else.