libaaruformat/docs/html/md_README.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-US">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=11"/>
<meta name="generator" content="Doxygen 1.18.0"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>libaaruformat: libaaruformat</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="dynsections.js"></script>
<script type="text/javascript" src="codefolding.js"></script>
<script type="text/javascript" src="clipboard.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript" src="cookie.js"></script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr id="projectrow">
  <td id="projectalign">
   <div id="projectname">libaaruformat<span id="projectnumber">&#160;1.0</span>
   </div>
   <div id="projectbrief">Aaru Data Preservation Suite - Format Library</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.18.0 -->
<script type="text/javascript">
let searchBox = new SearchBox("searchBox", "search/",'.html');
</script>
<script type="text/javascript">
document.addEventListener('DOMContentLoaded', codefold.init);
</script>
<script type="application/json" id="doxygen-config">
{
  "relPath": "",
  "generateTreeView": true,
  "searchEngine": true,
  "serverBasedSearch": false,
  "disableIndex": false,
  "dynamicMenus": true,
  "fullSidebar": false
}
</script>
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<div id="main-nav-mobile">
<div class="sm sm-dox"><input id="main-menu-state" type="checkbox"/>
<label class="main-menu-btn" for="main-menu-state">
<span class="main-menu-btn-icon"></span> Toggle main menu visibility</label>
<span id="searchBoxPos1"></span>
</div>
</div><!-- main-nav-mobile -->
<div id="main-nav">
  <ul class="sm sm-dox" id="main-menu">
    <li id="searchBoxPos2">
        <div id="MSearchBox" class="MSearchBoxInactive">
        <span class="left">
          <span id="MSearchSelect" class="search-icon"><span class="search-icon-dropdown"></span></span>
          <input type="text" id="MSearchField" value="" placeholder="Search" accesskey="S"/>
          </span><span class="right">
            <a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><div id="MSearchCloseImg" class="close-icon"></div></a>
          </span>
        </div>
    </li>
  </ul>
</div><!-- main-nav -->
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
  <div id="nav-tree">
    <div id="nav-tree-contents">
      <div id="nav-sync" class="sync"></div>
    </div>
  </div>
  <div id="splitbar" class="ui-resizable-handle">
  </div>
</div>
<script type="text/javascript">
document.addEventListener('DOMContentLoaded',() => { initNavTree('md_README.html','',''); });
</script>
<div id="container">
<div id="doc-content">
<!-- window showing the filter options -->
<div id="MSearchSelectWindow">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<div id="MSearchResults">
<div class="SRPage">
<div id="SRIndex">
<div id="SRResults"></div>
<div class="SRStatus" id="Loading">Loading...</div>
<div class="SRStatus" id="Searching">Searching...</div>
<div class="SRStatus" id="NoMatches">No Matches</div>
</div>
</div>
</div>
</div>

<div><div class="header">
  <div class="headertitle"><div class="title">libaaruformat </div></div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p><a class="anchor" id="autotoc_md0"></a></p>
<p><b>libaaruformat</b> is a C library for reading and writing <a href="https://github.com/aaru-dps/Aaru">AaruFormat</a> disk images — the native image format of the <a href="https://github.com/aaru-dps/Aaru">Aaru Data Preservation Suite</a>.</p>
<h1 class="doxsection"><a class="anchor" id="autotoc_md1"></a>
What is Aaru?</h1>
<p><a href="https://github.com/aaru-dps/Aaru">Aaru</a> is an open-source application for managing and preserving digital storage media. It can create, convert, compare, and analyze disk images from a wide variety of physical formats — floppy disks, hard drives, optical discs, tapes, memory cards, and more. Aaru is written in C# and runs on Windows, macOS, and Linux.</p>
<h1 class="doxsection"><a class="anchor" id="autotoc_md2"></a>
What is AaruFormat?</h1>
<p>AaruFormat is Aaru's own disk image file format, designed from the ground up for long-term archival and data preservation. Unlike most existing image formats, which target a single media family or prioritize emulation speed, AaruFormat aims to be a single, universal, extensible standard capable of faithfully representing any kind of digital or analog storage media — from punch cards to Blu-ray discs to magnetic flux captures.</p>
<p>Key design goals of the format include:</p>
<ul>
<li><b>Universality</b>: one format for all media types, eliminating the need for a different format per storage technology.</li>
<li><b>Completeness</b>: stores not just raw sector data, but also per-sector metadata (subchannel data, error flags, tags), per-media metadata (TOCs, catalog numbers, serial numbers), dump hardware information, and CHS geometry.</li>
<li><b>Compression</b>: blocks of sectors are compressed individually (LZMA, Zstd, or FLAC for audio), allowing random access without decompressing the entire image.</li>
<li><b>Deduplication</b>: a Deduplication Table (DDT) maps each sector to a content-addressed block via xxHash, so identical sectors across the image are stored only once.</li>
<li><b>Integrity</b>: every block carries a CRC64 checksum. Hashes (MD5, SHA1, SHA256, SpamSum, BLAKE3) are computed during writing and stored in the image.</li>
<li><b>Resilience</b>: optional Reed-Solomon erasure coding protects data blocks, deduplication tables, metadata, and the index against corruption, with transparent recovery on read.</li>
<li><b>Openness</b>: the format specification is freely available in <span class="tt">docs/spec/</span> and is published under the same terms as the library.</li>
</ul>
<h1 class="doxsection"><a class="anchor" id="autotoc_md3"></a>
What is libaaruformat?</h1>
<p><b>libaaruformat</b> is the reference implementation of AaruFormat, written in C. Aaru itself uses libaaruformat via P/Invoke, having replaced its earlier C# implementation. The library can also be used independently by other projects — emulators, forensic tools, archival systems, or anything written in C or a language with C FFI — to read and write AaruFormat images without depending on the .NET runtime.</p>
<p>The library is written in C99, has no external runtime dependencies (all third-party code is statically linked from the <span class="tt">3rdparty/</span> directory), and builds on macOS, Linux, and Windows. It is licensed under the LGPL-2.1.</p>
<h1 class="doxsection"><a class="anchor" id="autotoc_md4"></a>
Features</h1>
<h2 class="doxsection"><a class="anchor" id="autotoc_md5"></a>
Core</h2>
<ul>
<li>AaruFormat V1 images reading (writing will never be implemented)</li>
<li>AaruFormat V2 images reading and writing</li>
<li>Image identification and verification</li>
<li>LRU block cache for read performance</li>
<li>Large file (&gt;2 GiB) I/O support</li>
<li>Feature parity with the C# implementation</li>
</ul>
<h2 class="doxsection"><a class="anchor" id="autotoc_md6"></a>
Compression</h2>
<ul>
<li>LZMA</li>
<li>Zstandard (Zstd)</li>
<li>FLAC (for CD audio tracks)</li>
<li>Claunia Subchannel Transform (for CD subchannel data)</li>
</ul>
<h2 class="doxsection"><a class="anchor" id="autotoc_md7"></a>
Checksums &amp; Hashing</h2>
<ul>
<li>Hashing while writing: MD5, SHA1, SHA256, SpamSum, BLAKE3</li>
<li>CRC64 with SIMD acceleration (CLMUL on x86, VMULL on ARM)</li>
<li>ECC CD checksums</li>
</ul>
<h2 class="doxsection"><a class="anchor" id="autotoc_md8"></a>
Erasure Coding &amp; Data Recovery</h2>
<ul>
<li>Reed-Solomon error correction with configurable K,M parameters</li>
<li>GF(2^8) arithmetic with 4-path SIMD dispatch (AVX2, SSSE3, ARM NEON, scalar fallback)</li>
<li>Five protection groups: data blocks, DDT-primary, DDT-secondary, metadata, index</li>
<li>Transparent recovery on read</li>
<li>Recovery footer at EOF with backup header</li>
</ul>
<h2 class="doxsection"><a class="anchor" id="autotoc_md9"></a>
Console Disc Encryption/Decryption</h2>
<ul>
<li><b>PS3</b>: Per-sector AES-128-CBC encryption/decryption, encryption map support, IRD file parsing</li>
<li><b>Wii U</b>: Two-tier encryption (disc key + per-title keys), WUD/WUX format support</li>
<li><b>NGC/Wii</b>: Wii partition encryption, GameCube/Wii LFG PRNG junk detection and generation</li>
</ul>
<h2 class="doxsection"><a class="anchor" id="autotoc_md10"></a>
Media &amp; Metadata</h2>
<ul>
<li>Optical disc tracks</li>
<li>Tape files and partitions</li>
<li>Raw flux transition captures (Kryoflux, Pauline, Applesauce)</li>
<li>Apple Lisa tag handling</li>
<li>Dump hardware lists</li>
<li>CHS geometry retrieval and setting</li>
<li>XML metadata retrieval (writing will never be implemented)</li>
<li>JSON metadata retrieval and writing</li>
<li>Deduplication (xxHash-based with hash maps)</li>
<li>Media type list in sync with Aaru</li>
</ul>
<h2 class="doxsection"><a class="anchor" id="autotoc_md11"></a>
CLI Tool (<span class="tt">aaruformattool</span>)</h2>
<p>Optional command-line tool built with <span class="tt">-DBUILD_TOOL=ON</span>. Commands include:</p><ul>
<li><span class="tt">identify</span> — Identify AaruFormat version</li>
<li><span class="tt">info</span> — Display detailed image information</li>
<li><span class="tt">read</span> / <span class="tt">read_long</span> — Read sectors</li>
<li><span class="tt">verify</span> / <span class="tt">verify_sectors</span> — Verify image and sector integrity</li>
<li><span class="tt">compare</span> — Compare images</li>
<li><span class="tt">convert</span> — Convert between image formats</li>
<li><span class="tt">convert-ps3</span> — Convert PS3 disc images</li>
<li><span class="tt">convert-wiiu</span> — Convert Wii U disc images (WUD/WUX)</li>
<li><span class="tt">convert-ngcw</span> — Convert GameCube/Wii disc images</li>
<li><span class="tt">inject-media-tag</span> — Add media tags to images</li>
</ul>
<h2 class="doxsection"><a class="anchor" id="autotoc_md12"></a>
Platform Support</h2>
<ul>
<li>macOS (x86_64, arm64)</li>
<li>Linux (x86_64, aarch64, armv7, mips, riscv64)</li>
<li>Windows (MSVC, MinGW — x86, x64, ARM, ARM64)</li>
</ul>
<h2 class="doxsection"><a class="anchor" id="autotoc_md13"></a>
Developer</h2>
<ul>
<li>Unit testing (Google Test)</li>
<li>Automatic API documentation generation (Doxygen)</li>
<li>Format specification in AsciiDoc (<span class="tt">docs/spec/</span>)</li>
</ul>
<h1 class="doxsection"><a class="anchor" id="autotoc_md14"></a>
Things still to be implemented</h1>
<ul>
<li>Automatic media type generation from C# enumeration</li>
<li>NuGet package for linking with Aaru</li>
<li>Snapshots</li>
<li>Parent images</li>
<li>Data positioning measurements</li>
</ul>
<h1 class="doxsection"><a class="anchor" id="autotoc_md15"></a>
Building and Testing</h1>
<h2 class="doxsection"><a class="anchor" id="autotoc_md16"></a>
Standard Build</h2>
<div class="fragment"><div class="line">mkdir build</div>
<div class="line">cd build</div>
<div class="line">cmake ..</div>
<div class="line">cmake --build .</div>
</div><!-- fragment --><h2 class="doxsection"><a class="anchor" id="autotoc_md17"></a>
Running Tests</h2>
<div class="fragment"><div class="line">cd build</div>
<div class="line">ctest --verbose</div>
</div><!-- fragment --><h2 class="doxsection"><a class="anchor" id="autotoc_md18"></a>
Building with Address Sanitizer</h2>
<p>For debugging memory issues, you can build with Address Sanitizer enabled:</p>
<div class="fragment"><div class="line">mkdir build-asan</div>
<div class="line">cd build-asan</div>
<div class="line">cmake -DUSE_ASAN=ON -DCMAKE_BUILD_TYPE=Debug ..</div>
<div class="line">cmake --build .</div>
<div class="line">ctest --verbose</div>
</div><!-- fragment --><p>For detailed information on using Address Sanitizer to detect memory issues, see docs/ASAN_USAGE.md.</p>
<h2 class="doxsection"><a class="anchor" id="autotoc_md19"></a>
Build Options</h2>
<ul>
<li><span class="tt">-DBUILD_SHARED_LIBS=OFF</span> — Build as static library (default: shared)</li>
<li><span class="tt">-DBUILD_TOOL=ON</span> — Build the CLI tool (requires Argtable3, ICU, curses)</li>
<li><span class="tt">-DUSE_SLOG=ON</span> — Enable slog logging for debugging</li>
<li><span class="tt">-DUSE_ASAN=ON</span> — Enable Address Sanitizer for memory error detection </li>
</ul>
</div></div><!-- contents -->
</div><!-- PageDoc -->
</div><!-- doc-content -->
<div id="page-nav" class="page-nav-panel">
<div id="page-nav-resize-handle"></div>
<div id="page-nav-tree">
<div id="page-nav-contents">
</div><!-- page-nav-contents -->
</div><!-- page-nav-tree -->
</div><!-- page-nav -->
</div><!-- container -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
  <ul>
    <li class="footer">Generated by <a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.18.0 </li>
  </ul>
</div>
</body>
</html>