From a45d126383156b26fb615188a58d1a2e5b5eeffe Mon Sep 17 00:00:00 2001 From: Natalia Portillo Date: Thu, 15 Sep 2022 18:29:17 +0100 Subject: [PATCH] Add index continuation and track layout blocks. --- docs/Specification.fodt | 2535 +++++++++++++++++++++++---------------- 1 file changed, 1502 insertions(+), 1033 deletions(-) diff --git a/docs/Specification.fodt b/docs/Specification.fodt index 93ce5a6..5018987 100644 --- a/docs/Specification.fodt +++ b/docs/Specification.fodt @@ -1,24 +1,24 @@ - Natalia Portillo2012-08-11T17:46:302022-09-05T11:56:15.956112899P1DT18H27M44S91LibreOffice/7.4.0.3$Linux_X86_64 LibreOffice_project/f85e47c08ddd19c015c0114a68350214f7066f5a + Natalia Portillo2012-08-11T17:46:302022-09-15T18:28:51.785542293P1DT18H38M48S94LibreOffice/7.4.0.3$Linux_X86_64 LibreOffice_project/40$Build-3 - 394970 + 0 0 - 48844 - 21645 + 67153 + 32175 true false view2 - 6249 - 409958 + 15265 + 23715 0 - 394970 - 48842 - 416613 + 0 + 67151 + 32173 0 0 false @@ -91,7 +91,7 @@ true true - 10568947 + 10689362 true false @@ -163,14 +163,14 @@ - - + + - + @@ -183,18 +183,18 @@ - + - + - + @@ -210,92 +210,92 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -316,37 +316,37 @@ - - + + - - + + - + - + - + - + - + @@ -357,929 +357,1010 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -1323,7 +1404,7 @@ - + @@ -4504,28 +4585,28 @@ - + - + - + - + @@ -5500,775 +5581,879 @@ - - - - + + + + + + - - - - + + + + + + - - - - + + + + + + - + - + - - + + - - + + - + - + + + + + + + + + + + + + + + + - + - + - + - + - - - - - - - - - - + - + + + + + + + + + + - - - - - - - - - - - - - + - + - + - + - + - + - + - + - + - + + + + + + + + + - - - - - - + + + + + + + + + + - - - - - - - - - - - + + + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - + - - + + - - + + @@ -6280,11 +6465,11 @@ - AaruFormat Specification version 2.0 DRAFT 1 + AaruFormat Specification version 2.0 DRAFT 3 DRAFT VERSION, ALL STRUCTURES SUBJECT TO CHANGE, DO NOT IMPLEMENT THIS SPECIFICATION - AaruFormat Specification version 2.0 DRAFT 1 + AaruFormat Specification version 2.0 DRAFT 3 DRAFT VERSION, ALL STRUCTURES ARE SUBJECT TO CHANGE, DO NOT IMPLEMENT THIS SPECIFICATION @@ -6302,7 +6487,7 @@ - + iVBORw0KGgoAAAANSUhEUgAAA14AAAQ4CAYAAAAZ0g07AAAACXBIWXMAAApWAAAKVgF1tN0E AAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgABdOpJREFUeNrsnQd4Fce5 sEcdIXqvogmBhCTUO4jeQfQieu+9GBuDcWxsHDdsjBt2jG1sY3ClniNIyE2/sdOc5KbH6XGN @@ -8076,35 +8261,35 @@ gg== - AaruFormat Specification - Version 2.0 DRAFT 2 - 4th September 2022 + AaruFormat Specification + Version 2.0 DRAFT 3 + 15th September 2022 - AaruFormat Specification - Copyright © 2022 Natalia Portillo - The information in this document is subject to the following license. - You can use, distribute, redistribute, modify, collaborate, copy, send, implement, translate or talk about this specification as long as you comply with the following requirements: + AaruFormat Specification + Copyright © 2022 Natalia Portillo + The information in this document is subject to the following license. + You can use, distribute, redistribute, modify, collaborate, copy, send, implement, translate or talk about this specification as long as you comply with the following requirements: - You must retain the above copyright notice. + You must retain the above copyright notice. - You cannot change or modify the license of this specification. + You cannot change or modify the license of this specification. - There is only one official version of this specification, signed al so by Natalia Portillo's cryptographic key. + There is only one official version of this specification, signed al so by Natalia Portillo's cryptographic key. - Any modification you do to this specification is by nature, not official, and so you cannot pretend it is. + Any modification you do to this specification is by nature, not official, and so you cannot pretend it is. - You're invited to submit your modifications to appear in a future official revision of this specification, by sending it to the Aaru team. However you're not required to do so. + You're invited to submit your modifications to appear in a future official revision of this specification, by sending it to the Aaru team. However you're not required to do so. - Natalia Portillo or any member or collaborator of the Aaru team can take your modifications and made them appear in a future official revision of this specification without your explicit authorization. + Natalia Portillo or any member or collaborator of the Aaru team can take your modifications and made them appear in a future official revision of this specification without your explicit authorization. - You must add a “Portions ©” notice on your modifications. That notice will also must be retained. + You must add a “Portions ©” notice on your modifications. That notice will also must be retained. @@ -8118,7 +8303,7 @@ - Date + Date Version @@ -8136,10 +8321,10 @@ - 08 May 2022 + 08 May 2022 - 1.0 + 1.0 Official @@ -8153,10 +8338,10 @@ - 18 May 2022 + 18 May 2022 - 2.0d1 + 2.0d1 Draft @@ -8172,7 +8357,7 @@ - Sep 4 + 04 Sep 2022 2.0d2 @@ -8187,6 +8372,25 @@ Add flux data definitions. + + + 15 Sep 2022 + + + 2.0d3 + + + Draft + + + Natalia Portillo + + + Fix some typos. + Add index continuation block. + Add track layout block. + + @@ -8298,44 +8502,47 @@ Index entries10 The index block version 2 ('IDX2')11 Index entries11 - The data block ('DBLK')12 - The deduplication table ('DDT*') *DEPRECATED*13 - The deduplication table ('DDT2')14 - The geometry block ('GEOM')15 - The metadata block ('META')16 - The tracks block ('TRKS')17 - Track entries17 - Track types17 - The CICM XML metadata block ('CICM')18 - The checksum block ('CKSM')19 - Checksum entries19 - Checksum types19 - The data position measurement block ('DPM*')20 - The snapshot block ('SNAP')21 - The parent file block ('PRNT')22 - The dump hardware block ('DMP*')23 - Dump hardware entries23 - Dump hardware extents23 - The tape file block ('TFLE')24 - Tape file entries24 - The tape partition block ('TPBT')25 - Tape file entries25 - The compact disc indexes block ('CDIX')26 - Compact disc index entries26 - The flux data block ('FLUX')27 - Flux header entries27 - Annex A. Media types28 - Annex B. Data types29 - Annex C. Compression types32 - Annex D. Claunia Subchannel Transform33 - Annex E. Old Media types *DEPRECATED*34 + The index block continuation ('IDXC')12 + The data block ('DBLK')13 + The deduplication table ('DDT*') *DEPRECATED*14 + The deduplication table ('DDT2')15 + The geometry block ('GEOM')16 + The metadata block ('META')17 + The tracks block ('TRKS')18 + Track entries18 + Track types18 + The CICM XML metadata block ('CICM')19 + The checksum block ('CKSM')20 + Checksum entries20 + Checksum types20 + The data position measurement block ('DPM*')21 + The snapshot block ('SNAP')22 + The parent file block ('PRNT')23 + The dump hardware block ('DMP*')24 + Dump hardware entries24 + Dump hardware extents24 + The tape file block ('TFLE')25 + Tape file entries25 + The tape partition block ('TPBT')26 + Tape partition entries26 + The compact disc indexes block ('CDIX')27 + Compact disc index entries27 + The flux data block ('FLUX')28 + Flux header entries28 + The track layout block ('TKLY')29 + Sector mapping entries29 + Annex A. Media types30 + Annex B. Data types31 + Annex C. Compression types34 + Annex D. Claunia Subchannel Transform35 + Annex E. Old Media types *DEPRECATED*36 1. Introduction - This document is the detailed specification of AaruFormat. + This document is the detailed specification of AaruFormat. Audience - This specification is directed to emulator developers, software preservators, archives, museums and collectors, that want to have a common file format where to store, archive and manage, dumps and copies of any type of computer storage. + This specification is directed to emulator developers, software preservators, archives, museums and collectors, that want to have a common file format where to store, archive and manage, dumps and copies of any type of computer storage. Scope The scope of this specification is to define an open, free and universal file format able to store and describe any kind of digital or analog storage media for computer systems, in a clear and extensible way that allows for new media to be easily added, along with any kind of metadata describing them, plus verification and recovery data. Currently the idea is for it to be able to store punch cards, disks (magnetic, optical, magnetoptical) and tapes (analog and digital tapes), decoded or as audio tones and as magnetic or optical fluxes, with any kind of copy protection or absence of it. @@ -8343,21 +8550,21 @@ There are other formats pretending to achieve some of these goals, and precisely that's why this format is designed. To be a single, universal, extensible, standard, eliminating the need to use a different format for each type of storage. 2. Definitions Types - All binary types used in this specification are stored as little-endian values on the file. This specification follows the C syntax to denote hexadecimal values, and requires the reader have some knowledge on programming. + All binary types used in this specification are stored as little-endian values on the file. This specification follows the C syntax to denote hexadecimal values, and requires the reader have some knowledge on programming. Endianness - Unless otherwise specified, all fields in this specification are considered to be in Little-Endian format, that is the hexadecimal number 0x12345678 is stored in disk as the following sequence of bytes: 0x78 0x56 0x34 0x12. + Unless otherwise specified, all fields in this specification are considered to be in Little-Endian format, that is the hexadecimal number 0x12345678 is stored in disk as the following sequence of bytes: 0x78 0x56 0x34 0x12. Header identifiers - Header identifiers are 4 ASCII characters stored as a sequence of bytes inside a single 32 bits pack. They are shown in this specification enclosed in single quotes. For example, the header identifier 'AARU' should be stored on disk as 0x41 0x41 0x52 0x55. + Header identifiers are 4 ASCII characters stored as a sequence of bytes inside a single 32 bits pack. They are shown in this specification enclosed in single quotes. For example, the header identifier 'AARU' should be stored on disk as 0x41 0x41 0x52 0x55. Integers Integer values are designated in this specification started with a single caps letter indicating if signed or unsigned (S or U), continuing with Int and ending with the number of bits able to be stored in them. - That so, the signed integers should be: SInt8, SInt16, SInt32, Sint64 and SInt128. - And the unsigned integers should be: UInt8, UInt16, UInt32, UInt64, UInt128. + That so, the signed integers should be: SInt8, SInt16, SInt32, Sint64 and SInt128. + And the unsigned integers should be: UInt8, UInt16, UInt32, UInt64, UInt128. Strings - All strings are stored as a sequence of bytes, in Unicode's UTF-16 little endian encoding and terminated and filled with NULL (0x00) bytes. - String8 values mean the string is stored in Unicode’s UTF-8 encoding and terminated and filled with NULL (0x00) bytes. - StringA values mean the string is stored in ASCII encoding and terminated and filled with NULL (0x00) bytes. + All strings are stored as a sequence of bytes, in Unicode's UTF-16 little endian encoding and terminated and filled with NULL (0x00) bytes. + String8 values mean the string is stored in Unicode’s UTF-8 encoding and terminated and filled with NULL (0x00) bytes. + StringA values mean the string is stored in ASCII encoding and terminated and filled with NULL (0x00) bytes. Timestamp - All timestamps used in this specification are stored as a signed 64bit integer (SInt64) counting the number of nanoseconds in the UTC timezone after/before the epoch of 1st January 1601 at 00:00 of the Gregorian Calendar. This epoch is chosen because it is when the leap-year scheme was adopted. + All timestamps used in this specification are stored as a signed 64bit integer (SInt64) counting the number of nanoseconds in the UTC timezone after/before the epoch of 1st January 1601 at 00:00 of the Gregorian Calendar. This epoch is chosen because it is when the leap-year scheme was adopted. Media tag A media tag is a piece of data that is physically present in the media but it’s not part of the user data. It can be the table of contents, some manufacturing information, sector replacement tables, etc. Sector tag @@ -8365,7 +8572,7 @@ NULL NULLs are 0x00 bytes. 3. The Header - The master header is the most important part of an AaruFormat file. It describes the size and version of the file, as well as the contents and file-wide metadata. + The master header is the most important part of an AaruFormat file. It describes the size and version of the file, as well as the contents and file-wide metadata. @@ -8392,13 +8599,13 @@ UInt64 - 8 bytes + 8 bytes identifier - The header identifier, always 'AARUFRMT'. Old images can be found with ‘DICMFRMT’ but new ones must never be created with this value. + The header identifier, always 'AARUFRMT'. Old images can be found with ‘DICMFRMT’ but new ones must never be created with this value. @@ -8406,7 +8613,7 @@ String - 32 bytes + 32 bytes application @@ -8420,7 +8627,7 @@ UInt8 - 1 byte + 1 byte imageMajorVersion @@ -8434,7 +8641,7 @@ UInt8 - 1 byte + 1 byte imageMinorVersion @@ -8445,10 +8652,10 @@ - UInt8 + UInt8 - 1 byte + 1 byte applicationMajorVersion @@ -8459,10 +8666,10 @@ - UInt8 + UInt8 - 1 byte + 1 byte applicationMinorVersion @@ -8476,13 +8683,13 @@ MediaType - 4 bytes + 4 bytes mediaTypeOld - Type of media contained in this image. See Annex E.This field is only used if major version is less than or equal to 2. Reserved otherwise + Type of media contained in this image. See Annex E.This field is only used if major version is less than or equal to 2. Reserved otherwise @@ -8490,7 +8697,7 @@ UInt64 - 8 bytes + 8 bytes indexOffset @@ -8501,10 +8708,10 @@ - SInt64 + SInt64 - 8 bytes + 8 bytes creationTime @@ -8515,10 +8722,10 @@ - SInt64 + SInt64 - 8 bytes + 8 bytes lastWrittenTime @@ -8551,10 +8758,10 @@ - 4. The Blocks + 4. The Blocks The blocks in AaruFormat are the construction pieces of the file, that contains the data and metadata from the media represented by it. - The index block ('INDX') *DEPRECATED* - The index block contains pointers to all the blocks present on the file. It consists in the header followed by the number of entries indicated by the header. + The index block ('INDX') *DEPRECATED* + The index block contains pointers to all the blocks present on the file. It consists in the header followed by the number of entries indicated by the header. There can be several index blocks in a file, representing points in the past of the file, but only the last one shall be the one pointed by the main header. This block is deprecated and new images should not contain it. @@ -8589,15 +8796,15 @@ identifier - The index block identifier, always 'INDX' + The index block identifier, always 'INDX' - UInt16 + UInt16 - 2 bytes + 2 bytes entries @@ -8654,18 +8861,18 @@ blockType - The type of block this entry points to. + The type of block this entry points to. - UInt32 + UInt32 - 4 bytes + 4 bytes - dataType + dataType The type of data the block pointed by this entry contains. @@ -8688,10 +8895,10 @@ - - The index block version 2 ('IDX2') - The index block contains pointers to all the blocks present on the file. It consists in the header followed by the number of entries indicated by the header. - There can be several index blocks in a file, representing points in the past of the file, but only the last one shall be the one pointed by the main header. + + The index block version 2 ('IDX2') + The index block contains pointers to all the blocks present on the file. It consists in the header followed by the number of entries indicated by the header. + There can be several index blocks in a file, representing points in the past of the file, but only the last one shall be the one pointed by the main header. @@ -8700,63 +8907,63 @@ - Type + Type - Size + Size - Name + Name - Description + Description - UInt32 + UInt32 - 4 bytes + 4 bytes - identifier + identifier - The index block identifier, always 'INDX' + The index block identifier, always 'IDX2' - UInt64 + UInt64 - 8 bytes + 8 bytes - entries + entries - The number of entries following this header + The number of entries following this header - UInt64 + UInt64 - 8 bytes + 8 bytes - crc64 + crc64 - CRC64-ECMA checksum of the entries following this header + CRC64-ECMA checksum of the entries following this header - Index entries + Index entries @@ -8765,65 +8972,148 @@ - Type + Type - Size + Size - Name + Name - Description + Description - UInt32 + UInt32 - 4 bytes + 4 bytes - blockType + blockType - The type of block this entry points to. + The type of block this entry points to. - UInt32 + UInt32 - 4 bytes + 4 bytes - dataType + dataType - The type of data the block pointed by this entry contains. + The type of data the block pointed by this entry contains. - UInt64 + UInt64 - 8 bytes + 8 bytes - offset + offset - The offset in bytes from the start of the file where the block pointed by this entry starts. + The offset in bytes from the start of the file where the block pointed by this entry starts. - - The data block ('DBLK') - The data block contains data from the media. + The index block continuation ('IDXC') + The index block continuation behaves exactly as a the index block version 2 except it contains a pointer to the previous index block or index block continuation. + Only a single index block continuation must appear in any index block of any kind. + The intention of this block is to allow an index to be constructed before the image has been closed, so every time a new block has been written to an image it can be indexed, and in the case of an application crash, a partial image can be reconstructed using the written indexes. + This block is followed by index entries in the same format as present in the index block version 2. + + + + + + + + + Type + + + Size + + + Name + + + Description + + + + + + UInt32 + + + 4 bytes + + + identifier + + + The index block identifier, always 'IDXC' + + + + + UInt64 + + + 8 bytes + + + entries + + + The number of entries following this header + + + + + UInt64 + + + 8 bytes + + + crc64 + + + CRC64-ECMA checksum of the entries following this header + + + + + UInt64 + + + 8 bytes + + + previous + + + Pointer in image file to previous index block + + + + + The data block ('DBLK') + The data block contains data from the media. It consists of a header followed by the compressed, or uncompressed, data. @@ -8857,15 +9147,15 @@ identifier - The data block identifier, always 'DBLK' + The data block identifier, always 'DBLK' - UInt16 + UInt16 - 2 bytes + 2 bytes type @@ -8876,10 +9166,10 @@ - UInt16 + UInt16 - 2 bytes + 2 bytes compression @@ -8961,11 +9251,11 @@ A data block can contain user data, that is, media sectors, or separate data, for example media or sector tags. - In case a single data block contains several items (e.g. sectors or sector tags), the sectorSize frield indicates the size in bytes of each individual item. In case it contains a single item (e.g. media tags), sectorSize must be set to 0. - - The deduplication table ('DDT*') *DEPRECATED* - The deduplication table is an array of pointers laid out sequentially. Each sector on the media represents a pointer in the deduplication table. At least one deduplication table with a data type of UserData must be present in the image. - This block is deprecated and new images must not contain it. + In case a single data block contains several items (e.g. sectors or sector tags), the sectorSize frield indicates the size in bytes of each individual item. In case it contains a single item (e.g. media tags), sectorSize must be set to 0. + + The deduplication table ('DDT*') *DEPRECATED* + The deduplication table is an array of pointers laid out sequentially. Each sector on the media represents a pointer in the deduplication table. At least one deduplication table with a data type of UserData must be present in the image. + This block is deprecated and new images must not contain it. @@ -8998,35 +9288,35 @@ identifier - The deduplication table identifier, always 'DDT*' + The deduplication table identifier, always 'DDT*' - UInt16 + UInt16 - 2 bytes + 2 bytes type - The data type pointed by this table. See Annex B. + The data type pointed by this table. See Annex B. - UInt16 + UInt16 - 2 bytes + 2 bytes compression - The compression algorithm used in the table. See Annex C. + The compression algorithm used in the table. See Annex C. @@ -9045,10 +9335,10 @@ - UInt64 + UInt64 - 8 bytes + 8 bytes entries @@ -9068,7 +9358,7 @@ cmpLength - The size in bytes of the compressed table that follows this header. + The size in bytes of the compressed table that follows this header. @@ -9082,7 +9372,7 @@ length - The size in bytes of the table block when decompressed. + The size in bytes of the table block when decompressed. @@ -9096,7 +9386,7 @@ cmpCrc64 - The CRC64-ECMA checksum of the compressed table that follows this header. + The CRC64-ECMA checksum of the compressed table that follows this header. @@ -9110,15 +9400,15 @@ crc64 - The CRC64-ECMA checksum of the decompressed table. + The CRC64-ECMA checksum of the decompressed table. - Each entry in the table points to a data block and to a specific item contained inside it. + Each entry in the table points to a data block and to a specific item contained inside it. Entry number 0 of the deduplication table points to the data belonging to LBA 0 of the media, and so on. - The entry is calculated as (byte offset of data block in file << shift) + item number in data block. Therefore a pointer with a raw value of 0x8003 in a deduplication table with a shift value of 5 indicates the third item in the data block located at absolute byte offset 1024 in the file. - A special case are the deduplication tables with data types CdSectorPrefixCorrected and CdSectorSuffixCorrected. In their case a mask of 0x00FFFFFF is used to obtain the pointer and a mask of 0xFF000000 is used to obtain the following flags: + The entry is calculated as (byte offset of data block in file << shift) + item number in data block. Therefore a pointer with a raw value of 0x8003 in a deduplication table with a shift value of 5 indicates the third item in the data block located at absolute byte offset 1024 in the file. + A special case are the deduplication tables with data types CdSectorPrefixCorrected and CdSectorSuffixCorrected. In their case a mask of 0x00FFFFFF is used to obtain the pointer and a mask of 0xFF000000 is used to obtain the following flags: @@ -9155,7 +9445,7 @@ 0x10000000 - The sector has not been dumped. Ignore the pointer. + The sector has not been dumped. Ignore the pointer. @@ -9166,7 +9456,7 @@ 0x20000000 - The suffix (only for MODE 1 sectors) or prefix is correct and can be regenerated. Ignore the pointer. + The suffix (only for MODE 1 sectors) or prefix is correct and can be regenerated. Ignore the pointer. @@ -9203,9 +9493,9 @@ - The deduplication table ('DDT2') - To be defined. - The geometry block ('GEOM') + The deduplication table ('DDT2') + To be defined. + The geometry block ('GEOM') The geometry block stores the geometry information of the media to allow for CHS<->LBA transformations. This information is not necessarily physically correct, but may be whatever translation the drive had when it was dumped. @@ -9240,15 +9530,15 @@ identifier - The geometry table identifier, always 'GEOM' + The geometry table identifier, always 'GEOM' - UInt32 + UInt32 - 4 bytes + 4 bytes cylinders @@ -9259,10 +9549,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes heads @@ -9273,22 +9563,22 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes sectorsPerTrack - The number of sectors per track. + The number of sectors per track. - - The metadata block ('META') + + The metadata block ('META') The metadata block stores information about the media, that is not data in the media itself, like manufacturer, model, sequence, etc. All strings in this block are stored as little-endian Unicode’s UTF-16 null-terminated. @@ -9322,15 +9612,15 @@ identifier - The metadata table identifier, always 'META' + The metadata table identifier, always 'META' - UInt32 + UInt32 - 4 bytes + 4 bytes blockSize @@ -9341,10 +9631,10 @@ - SInt32 + SInt32 - 4 bytes + 4 bytes mediaSequence @@ -9355,24 +9645,24 @@ - SInt32 + SInt32 - 4 bytes + 4 bytes lastMediaSequence - The number of sectors per track. + The number of sectors per track. - UInt32 + UInt32 - 4 bytes + 4 bytes creatorOffset @@ -9383,10 +9673,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes creatorLength @@ -9397,10 +9687,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes commentsOffset @@ -9411,10 +9701,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes commentsLength @@ -9425,10 +9715,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaTitleOffset @@ -9439,10 +9729,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaTitleLength @@ -9453,10 +9743,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaManufacturerOffset @@ -9467,10 +9757,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaManufacturerLength @@ -9481,10 +9771,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaModelOffset @@ -9495,10 +9785,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaModelLength @@ -9509,10 +9799,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaSerialNumberOffset @@ -9523,10 +9813,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaSerialNumberLength @@ -9537,10 +9827,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaBarcodeOffset @@ -9551,10 +9841,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaBarcodeLength @@ -9565,10 +9855,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaPartNumberOffset @@ -9579,10 +9869,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes mediaPartNumberLength @@ -9593,10 +9883,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes driveManufacturerOffset @@ -9607,10 +9897,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes driveManufacturerLength @@ -9621,10 +9911,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes driveModelOffset @@ -9635,10 +9925,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes driveModelLength @@ -9649,10 +9939,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes driveSerialNumberOffset @@ -9663,10 +9953,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes driveSerialNumberLength @@ -9677,10 +9967,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes driveFirmwareRevisionOffset @@ -9691,10 +9981,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes driveFirmwareRevisionLength @@ -9705,7 +9995,7 @@ - The tracks block ('TRKS') + The tracks block ('TRKS') The tracks block contains a list of tracks as defined in the table of contents, or equivalent structure, usually found in optical discs (like CD, DVD, etc). @@ -9739,15 +10029,15 @@ identifier - The tracks block identifier, always 'TRKS' + The tracks block identifier, always 'TRKS' - UInt16 + UInt16 - 2 bytes + 2 bytes entries @@ -9771,7 +10061,7 @@ - Track entries + Track entries @@ -9798,7 +10088,7 @@ UInt8 - 1 byte + 1 byte sequence @@ -9809,10 +10099,10 @@ - UInt8 + UInt8 - 1 byte + 1 byte type @@ -9906,7 +10196,7 @@ - Track types + Track types @@ -9992,9 +10282,9 @@ - - The CICM XML metadata block ('CICM') - This header indicates the presence of an embedded CICM XML metadata sidecar. It is stored as is without taking into consideration its contents. + + The CICM XML metadata block ('CICM') + This header indicates the presence of an embedded CICM XML metadata sidecar. It is stored as is without taking into consideration its contents. @@ -10027,15 +10317,15 @@ identifier - The CICM XML metadata table identifier, always 'CICM' + The CICM XML metadata table identifier, always 'CICM' - UInt32 + UInt32 - 4 bytes + 4 bytes length @@ -10047,8 +10337,8 @@ - - The checksum block ('CKSM') + + The checksum block ('CKSM') This block contains an array of checksums of the user data contained in this image. In the case of CompactDisc and similar media the checksums are applied to the full, including prefix and suffix, sector (2352 bytes). When an image is altered this block must be considered stale and removed or omitted from the latest index as appropriate. @@ -10083,29 +10373,29 @@ identifier - The tracks block identifier, always 'CKSM' + The tracks block identifier, always 'CKSM' - UInt32 + UInt32 - 4 bytes + 4 bytes length - The length in bytes of the data following this header. + The length in bytes of the data following this header. - UInt16 + UInt16 - 2 bytes + 2 bytes entries @@ -10115,7 +10405,7 @@ - Checksum entries + Checksum entries @@ -10142,7 +10432,7 @@ UInt8 - 1 byte + 1 byte type @@ -10153,10 +10443,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes length @@ -10166,7 +10456,7 @@ - Checksum types + Checksum types @@ -10241,20 +10531,20 @@ - - The data position measurement block ('DPM*') - The data position measurement block stores information about the physical structure of a disc my taking measurements on the position of each sector. + + The data position measurement block ('DPM*') + The data position measurement block stores information about the physical structure of a disc my taking measurements on the position of each sector. The format of this block is pending to be defined in a future version of this specification. - - The snapshot block ('SNAP') + + The snapshot block ('SNAP') The snapshot block contains a list of old indexes that are used as snapshots in time of old versions of the media contained by this image. The format of this block is pending to be defined in a future version of this specification. - The parent file block ('PRNT') + The parent file block ('PRNT') This block contains information needed to find the parent image file from which this one is derived. All sectors that are found as non-written in this image should be read on the parent image. The format of this block is pending to be defined in a future version of this specification. - The dump hardware block ('DMP*') + The dump hardware block ('DMP*') This block contains an array of hardware used to dump the media as well as a list of extents dumped by each one. @@ -10288,15 +10578,15 @@ identifier - The dump hardware block identifier, always 'DMP*' + The dump hardware block identifier, always 'DMP*' - UInt16 + UInt16 - 2 bytes + 2 bytes entries @@ -10307,24 +10597,24 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes length - The length in bytes of the data following this header. + The length in bytes of the data following this header. - UInt64 + UInt64 - 8 bytes + 8 bytes crc64 @@ -10334,7 +10624,7 @@ - Dump hardware entries + Dump hardware entries @@ -10358,10 +10648,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes manufacturerLength @@ -10372,10 +10662,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes modelLength @@ -10386,10 +10676,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes revisionLength @@ -10400,10 +10690,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes firmwareLength @@ -10414,10 +10704,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes serialLength @@ -10428,10 +10718,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes softwareNameLength @@ -10442,10 +10732,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes softwareVersionLength @@ -10456,10 +10746,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes softwareOperatingSystemLength @@ -10470,20 +10760,20 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes extents - How many extents are after the strings. + How many extents are after the strings. - Dump hardware extents + Dump hardware extents @@ -10507,10 +10797,10 @@ - UInt64 + UInt64 - 8 bytes + 8 bytes start @@ -10521,10 +10811,10 @@ - UInt64 + UInt64 - 8 bytes + 8 bytes end @@ -10537,7 +10827,7 @@ Each dump hardware entry is followed by the strings in the following order: manufacturer, model, revision, firmware version, serial number, software name, software version, software operating system. The extents follow the last written string. - The tape file block ('TFLE') + The tape file block ('TFLE') This block lists all tape files. Tape files are separations written to some media, usually sequentially written digital tapes, that are called “filemarks”. @@ -10571,15 +10861,15 @@ identifier - The tape file block identifier, always 'TFLE' + The tape file block identifier, always 'TFLE' - UInt16 + UInt16 - 2 bytes + 2 bytes entries @@ -10590,24 +10880,24 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes length - The length in bytes of the data following this header. + The length in bytes of the data following this header. - UInt64 + UInt64 - 8 bytes + 8 bytes crc64 @@ -10617,7 +10907,7 @@ - Tape file entries + Tape file entries @@ -10641,10 +10931,10 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes file @@ -10655,10 +10945,10 @@ - UInt8 + UInt8 - 1 byte + 1 byte partition @@ -10669,10 +10959,10 @@ - UInt64 + UInt64 - 8 bytes + 8 bytes firstBlock @@ -10683,10 +10973,10 @@ - UInt64 + UInt64 - 8 bytes + 8 bytes lastBlock @@ -10697,7 +10987,7 @@ - The tape partition block ('TPBT') + The tape partition block ('TPBT') This block lists all tape files. Tape partitions are separations written to some media. They’re used to separate two sets of data related but distant enough to be in the same tape. A famous example of using them is the LTFS filesystem. @@ -10731,15 +11021,15 @@ identifier - The tape partition block identifier, always 'TPBT' + The tape partition block identifier, always 'TPBT' - UInt16 + UInt16 - 2 bytes + 2 bytes entries @@ -10750,24 +11040,24 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes length - The length in bytes of the data following this header. + The length in bytes of the data following this header. - UInt64 + UInt64 - 8 bytes + 8 bytes crc64 @@ -10777,7 +11067,7 @@ - Tape file entries + Tape partition entries @@ -10801,10 +11091,10 @@ - UInt8 + UInt8 - 1 byte + 1 byte number @@ -10815,10 +11105,10 @@ - UInt64 + UInt64 - 8 bytes + 8 bytes firstBlock @@ -10829,10 +11119,10 @@ - UInt64 + UInt64 - 8 bytes + 8 bytes lastBlock @@ -10843,9 +11133,9 @@ - - The compact disc indexes block ('CDIX') - On CompactDisc and derived media, a track can have several indexes. They are used in some discs as a way of marking separations in the data, like different parts of a musical performance. + + The compact disc indexes block ('CDIX') + On CompactDisc and derived media, a track can have several indexes. They are used in some discs as a way of marking separations in the data, like different parts of a musical performance. The table of contents always points to index 1, and all other indexes, including 0 (the pregap), are found on the subchannel information. This block contains a list of all known indexes for fast lookup. @@ -10880,15 +11170,15 @@ identifier - The compact disc indexes block identifier, always 'CDIX' + The compact disc indexes block identifier, always 'CDIX' - UInt16 + UInt16 - 2 bytes + 2 bytes entries @@ -10899,24 +11189,24 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes length - The length in bytes of the data following this header. + The length in bytes of the data following this header. - UInt64 + UInt64 - 8 bytes + 8 bytes crc64 @@ -10926,7 +11216,7 @@ - Compact disc index entries + Compact disc index entries @@ -10950,10 +11240,10 @@ - UInt16 + UInt16 - 2 bytes + 2 bytes track @@ -10964,10 +11254,10 @@ - UInt16 + UInt16 - 2 bytes + 2 bytes index @@ -10978,10 +11268,10 @@ - SInt32 + SInt32 - 4 bytes + 4 bytes lba @@ -10992,12 +11282,12 @@ - The flux data block ('FLUX') - Several hardware devices exists that read magnetic media at flux transition level, such as the Kryoflux, Pauline and Applesauce. + The flux data block ('FLUX') + Several hardware devices exists that read magnetic media at flux transition level, such as the Kryoflux, Pauline and Applesauce. Flux transition level reads are by definition digital representations of analog properties of media, and as such cannot be reliably accessed on a sector by sector basis without further processing. As such, the data is accessed in a “capture” block, the size of which varies depending on the media and hardware used to image. As examples, for floppies, this capture is usually represented by a single revolution for a track, but might be larger for some hardware (e.g. Applesause captures 1¼ revolutions). For Quick Disks, the smallest capture will likely be one entire side of the media. Each capture block will contain two streams of flux data, one for the actual user data, and one for the indexing signal. - The flux data itself is represented by an array of unsigned bytes (UInt8), where each byte contains the number of ticks since the last flux transition. If no transition is registered inside of one byte worth of ticks, the byte will contain 0xFF and continue at the following byte with previous ticks added. - This block contains a list of all known flux captures. + The flux data itself is represented by an array of unsigned bytes (UInt8), where each byte contains the number of ticks since the last flux transition. If no transition is registered inside of one byte worth of ticks, the byte will contain 0xFF and continue at the following byte with previous ticks added. + This block contains a list of all known flux captures. @@ -11030,15 +11320,15 @@ identifier - The flux data block identifier, always 'FLUX' + The flux data block identifier, always 'FLUX' - UInt16 + UInt16 - 2 bytes + 2 bytes entries @@ -11049,10 +11339,10 @@ - UInt64 + UInt64 - 8 bytes + 8 bytes crc64 @@ -11062,7 +11352,7 @@ - Flux header entries + Flux header entries @@ -11086,35 +11376,35 @@ - UInt32 + UInt32 - 4 bytes + 4 bytes head - Head the data corresponds to. + Head the data corresponds to. - UInt16 + UInt16 - 2 bytes + 2 bytes track - Track the data corresponds to. + Track the data corresponds to. - UInt8 + UInt8 1 byte @@ -11123,15 +11413,15 @@ subtrack - Substep of a track that the data corresponds to. + Substep of a track that the data corresponds to. - UInt64 + UInt64 - 8 bytes + 8 bytes resolution @@ -11141,9 +11431,188 @@ - - Annex A. Media types - This is a list of all known media types as of writing this specification. This list is not to be considered complete, being the source of libaaruformat the origin of the most up-to-date list. + The track layout block ('TKLY') + This block describes the relationship between the physical tracks and the sectors as defined by the deduplication table. + As some magnetic media, including floppies and hard disks, can have complex layouts that do not easily translate into logical block addresses, as used by the deduplication table, this block allows us to find the proper data. + Each track layout block shall correspond to a separate (sub)track and head combination, and is followed by a number of sector mapping entries as described below. + Ideally, if known, the sectors shall be written ordered by their physical location, to preserve any possible interleaving information. Sector numbers can be duplicate. + This block must not be used in optical media or logically addressable block based media. + In the case the pointed LBA is marked as not dumped, and the flux block is present, it indicates the sector has not been decoded (not found, too damaged, etc) and is considered not dumped unless flags indicate other situation. + In the case there is a flux block for a particular (sub)track and there is not a corresponding track layout block for it, it means the whole (sub)track has not been decoded. + + + + + + + + + Type + + + Size + + + Name + + + Description + + + + + + UInt32 + + + 4 bytes + + + identifier + + + The flux data block identifier, always 'FLUX' + + + + + UInt64 + + + 8 bytes + + + crc64 + + + The CRC64-ECMA checksum of the data following this header + + + + + UInt32 + + + 4 bytes + + + head + + + Head the block corresponds to + + + + + UInt16 + + + 2 bytes + + + track + + + Track the block corresponds to + + + + + UInt8 + + + 1 byte + + + subtrack + + + Substep of a track the data corresponds to + + + + + UInt16 + + + 2 bytes + + + sectors + + + Number of sectors in this (sub)track, and therefore, number of entries following this header + + + + + UInt64 + + + 8 bytes + + + flux + + + Pointer to the flux data block that contains the flux information for this (sub)track + + + + Sector mapping entries + + + + + + + + + Type + + + Size + + + Name + + + Description + + + + + + UInt16 + + + 2 bytes + + + sector + + + Sector number as present in the appropriate media sector header or equivalent + + + + + UInt64 + + + 8 bytes + + + block + + + Position in the deduplication table this sector and its flags is stored + + + + + Annex A. Media types + This is a list of all known media types as of writing this specification. This list is not to be considered complete, being the source of libaaruformat the origin of the most up-to-date list. TO BE DEFINED @@ -11171,12 +11640,12 @@ 0 - Unknown media type + Unknown media type - Annex B. Data types - These are all the data types that can be contained in a data block or pointed by a deduplication table. They represent user data, media tags or sector tags. + Annex B. Data types + These are all the data types that can be contained in a data block or pointed by a deduplication table. They represent user data, media tags or sector tags. @@ -11857,7 +12326,7 @@ - Annex C. Compression types + Annex C. Compression types These are all compression types that can be used and found in AaruFormat images. @@ -11905,14 +12374,14 @@ - Annex D. Claunia Subchannel Transform + Annex D. Claunia Subchannel Transform The subchannel in a CompactDisc media, and derived, consists of 8 elements: P, Q, R, S, T, U, V, W. Usually they are interleaved, so each byte read from the media contains 1 single bit of each element. This makes the data appear much more random than it really is, making LZMA be really slow an inefficient (less than 2% of compression gains) when applied to the data as it. The transform is really simple, first all bits are de-interleaved so the 8 elements are individual bytes. Then all the P bytes from all the sectors are written sequentially before all Q bytes that are written before all R bytes and so on. While this takes the double amount of memory for the transformation (approximately 32MiB more of memory), the resulting data is compressed up to 10 times faster with a compression gain of approximately 96% (on discs that do not contain data in the R-W subchannels, as 99% of discs are). - Annex E. Old Media types *DEPRECATED* - This is a list of all known media types as of writing this specification. This list is not to be considered complete, being the source of libaaruformat the origin of the most up-to-date list. + Annex E. Old Media types *DEPRECATED* + This is a list of all known media types as of writing this specification. This list is not to be considered complete, being the source of libaaruformat the origin of the most up-to-date list. These values are deprecated and must not be used in new images, but can be found in old images.