Korean broadcasts use EUC-KR encoding (variable-width) in CEA-708
captions, where ASCII is 1 byte and Korean characters are 2 bytes.
The decoder was always writing 2 bytes per character (UTF-16BE style),
causing NULL bytes to be inserted before every ASCII character.
Changes:
- Add is_utf16_charset() to detect fixed-width 16-bit encodings
- Modify write_char() to accept use_utf16 flag:
- true: Always 2 bytes (UTF-16BE for Japanese, issue #1451)
- false: 1 byte for ASCII, 2 bytes for extended (EUC-KR for Korean)
- Detect charset type in write_row() before building output buffer
This fixes Korean subtitle extraction when using --service "1[EUC-KR]"
while maintaining compatibility with Japanese UTF-16BE (issue #1451).
Closes#1065🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When converting CEA-708 decoder settings from C to Rust via from_ctype(),
a null timing pointer would cause the entire conversion to fail and return
None. This triggered the unwrap_or(default()) fallback, resetting critical
settings like `enabled` and `services_enabled` to false/0.
This caused CEA-708 captions to not be extracted (exit code 10) even when
--service was specified, because the decoder's is_active flag was reset
to 0 during demuxer initialization.
The fix handles null timing pointer gracefully by using a default
CommonTimingCtx instead of propagating None, preserving the other
decoder settings.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit adds detection and basic handling of DVB teletext streams
in WTV (Windows TV) files. Previously, teletext streams were silently
ignored.
Changes:
- Add WTV_STREAM_TELETEXT GUID to wtv_constants.h
- Detect teletext streams by examining the format GUID at offset 0x4C
in MSTVCAPTION stream metadata
- Initialize teletext decoder when teletext stream is found
- Add timing support for teletext streams
- Wrap teletext data in PES headers for the teletext decoder
Limitation: WTV files store teletext in Microsoft's VBI sample format,
which differs from standard DVB teletext data units. The decoder will
process the data but may not extract subtitles from all WTV files.
This is noted in a warning message shown when teletext is detected.
Even FFmpeg's libzvbi fails to decode this format in the test sample.
Addresses: #1391🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
On Windows, when processing MP4/MOV files with CEA-708 captions, the
output file was being truncated to only the last subtitle. This occurred
because:
1. C code opened the file using open() and stored the fd in writer->fd
2. At end of processing, Rust's ccxr_flush_decoder was called
3. Rust checked writer->fhandle (a separate Windows-specific field)
4. Since fhandle was null (C only set fd), Rust called File::create()
5. File::create() truncates existing files, losing all previous content
The fix checks if fd is already valid before creating a new file. If fd
is valid, it converts it to a Windows handle using _get_osfhandle(),
avoiding the file truncation.
Fixes#1449🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Redirect stderr to /dev/null for the GPAC source file search to avoid
showing "No such file or directory" error when GPAC is not installed.
The build continues to work correctly in both cases.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes two buffer overflow vulnerabilities reported in issues #1427 and #1428:
- #1428 (Global buffer overflow in slice_header): The slice_type value
read from H.264 exp-golomb data was used to index slice_types[] array
without bounds checking. Valid values are 0-9 per H.264 spec Table 7-6.
Now validates slice_type < 10 before use.
- #1427 (Heap buffer overflow in parse_PMT): ES_info_length from PMT
descriptor data was trusted without validation against buffer bounds.
Malformed PMT with excessive ES_info_length could read past buffer end.
Now validates ES_info_length and descriptor lengths against buffer.
Both issues were discovered using AddressSanitizer with crafted TS files.
Fixes#1427Fixes#1428🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Homebrew installs leptonica as 'libleptonica.dylib', not 'liblept.dylib'.
Changed AC_CHECK_LIB from [lept] to [leptonica] to match the actual
library name on macOS.
The AC_CHECK_LIB checks in configure.ac need LDFLAGS and CPPFLAGS
to find libraries installed via Homebrew (in /opt/homebrew on Apple
Silicon or /usr/local on Intel Macs).
Fixes#1173 - Error in ./configure enabling hardsubx on Mac
Fixes#1306 - Add HARDSUBX compilation docs for macOS
The configure.ac script failed on macOS with "binary operator expected"
because pkg-config output was unquoted. When pkg-config returns multiple
libraries (e.g., "-ltesseract -lcurl"), the unquoted expansion caused
`test ! -z` to receive multiple arguments instead of a single string.
Changes:
- Quote pkg-config output in TESSERACT_PRESENT conditional (mac & linux)
- Add macOS section to docs/HARDSUBX.txt with all build methods
- Add GitHub Actions jobs to test HARDSUBX builds on macOS:
- build_shell_hardsubx: Tests ./build.command -hardsubx
- build_autoconf_hardsubx: Tests ./configure --enable-hardsubx
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The include "../lib_hash/sha2.h" in params.c requires an include path
that makes "../lib_hash" resolve to "thirdparty/lib_hash".
Changed -I../src/lib_hash (which doesn't exist) to
-I../src/thirdparty/lib_hash. With this path, the compiler searches
for "../lib_hash/sha2.h" as:
../src/thirdparty/lib_hash/../lib_hash/sha2.h
= ../src/thirdparty/lib_hash/sha2.h ✓
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Tests all three Dockerfile build types in parallel:
- minimal: Basic CCExtractor without OCR
- ocr: CCExtractor with Tesseract OCR support
- hardsubx: CCExtractor with burned-in subtitle extraction
Each job builds from local source and verifies the image works
by running --version. Uses GitHub Actions cache for faster rebuilds.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add a new `-system-libs` flag to mac/build.command that uses
system-installed libraries via pkg-config instead of bundled ones.
This enables Homebrew formula compatibility while preserving the
default standalone build behavior.
When `-system-libs` is passed:
- Uses pkg-config for: freetype2, gpac, libpng, libprotobuf-c,
libutf8proc, zlib
- Does not compile bundled thirdparty sources
- Links against system libraries
Default behavior (no flag):
- Compiles bundled libraries as before
- No change to existing builds
Also adds a CI job `build_shell_system_libs` to test the new flag.
Refs #1580, #1534🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously, configure would succeed even without GPAC installed,
leading to a confusing compile-time error:
"gpac/isomedia.h: No such file or directory"
Now configure checks for GPAC via pkg-config and fails early with
a helpful error message listing the package names for common distros:
- gpac-devel (Fedora/RHEL)
- libgpac-dev (Debian/Ubuntu)
- gpac (Arch)
Fixes#1584🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes#1550 - Docker builds were broken after PR #1535 switched from
vendored GPAC to system GPAC.
Changes:
- Switch from Alpine to Debian Bookworm (Alpine's musl libc has issues
with Rust bindgen's libclang dynamic loading)
- Support three build variants via BUILD_TYPE argument:
- minimal: No OCR support
- ocr (default): Tesseract OCR for bitmap subtitles
- hardsubx: OCR + FFmpeg for burned-in subtitle extraction
- Support dual source modes via USE_LOCAL_SOURCE argument:
- 0 (default): Clone from GitHub (standalone Dockerfile)
- 1: Use local source (faster for developers)
- Add .dockerignore to exclude build artifacts (~2.7GB -> ~900KB context)
- Update README.md with comprehensive build instructions
Tested all three variants successfully:
- minimal: ~130MB image
- ocr: ~215MB image
- hardsubx: ~610MB image
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously, when using -out=mcc with raw input files (-in=raw),
CCExtractor would print "Output format not supported" and produce
no output. This was because the raw file processing path decoded
CEA-608 data to text, but MCC format requires raw cc_data bytes.
The fix adds a new code path that bypasses the 608 decoder when
MCC output is requested:
- Added process_raw_for_mcc() helper function that:
- Converts 2-byte raw pairs to 3-byte cc_data format
- Wraps each CC pair in CDP format via mcc_encode_cc_data()
- Maintains proper timing at 29.97fps
- Modified raw_loop() to detect MCC output and use the new path
Test results with McPoodle raw files:
- Before: "Output format not supported" (exit code 10)
- After: Valid MCC file with proper timing and CDP-wrapped data
Fixes#1542🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes#1455
When read_video_pes_header() encounters a malformed or truncated PES
packet (returns -1), copy_capbuf_demux_data() previously returned
CCX_EOF which terminated the entire file processing. This was overly
aggressive - a single broken PES packet should be skipped, not
terminate the entire file.
UK Freeview DVB recordings from September 2022 onwards contain some
malformed PES packets in the DVB subtitle stream that triggered this
condition, causing ccextractor to stop at 0% with "Processing ended
prematurely" error even though VLC could display the subtitles.
The fix changes the error handling to skip the broken packet and
continue processing:
- Before: return CCX_EOF (terminates file)
- After: return CCX_OK (skips packet, continues)
Test results with UK Freeview sample:
- Before: 0% processed, 0 subtitles extracted
- After: 100% processed, 10 subtitles extracted correctly
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The HEVC NAL type constants are defined for completeness and reference,
but not all are currently used in the codebase.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
HEVC uses B-frames extensively, causing CC data to arrive in decode
order instead of presentation order. This was causing character pairs
to be scrambled (e.g., "MEDIOCRE" became "MIOEDCRE").
Changes:
- Implement PTS-based sequence numbering for HEVC CC data (similar to H.264)
- Change flush logic to only trigger on IDR frames (not every VCL NAL)
- Add HEVC fallback detection for streams without PAT/PMT
Fixes#1639 (ATSC 3.0 HEVC caption extraction)
Tested with issue_1639_sample.ts and caption_test_1690.ts
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes#1690 - Captions fail to extract on HEVC video stream
HEVC video streams with embedded EIA-608/708 captions weren't being
extracted, even though VLC/MPV could display them.
Root causes fixed:
1. HEVC stream type (0x24) wasn't recognized for CC extraction
2. HEVC NAL parsing used H.264 format (1-byte) instead of HEVC (2-byte)
3. HEVC SEI types (39/40) weren't handled (only H.264 SEI type 6)
4. CC data accumulation across SEIs caused u8 overflow/garbled output
Changes:
- C code: Add HEVC stream detection, CCX_HEVC buffer type, is_hevc flag
- Rust code: HEVC NAL header parsing (2-byte, type=(byte[0]>>1)&0x3F),
HEVC SEI handling (PREFIX_SEI=39, SUFFIX_SEI=40), immediate CC flush
Thanks to @trufio465-bot for the initial research in PR #1735.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The AVC parser would fail with "Leading bytes are non-zero" error when
processing HLS/Twitch stream segments that start mid-stream without
proper NAL unit headers at the beginning.
Root cause: When process_avc encountered non-zero leading bytes, it
returned an error with 0 bytes processed. The C code would not remove
any bytes from the buffer, causing subsequent data to accumulate with
the corrupt beginning, leading to infinite errors.
Fix:
- Add find_nal_start_code() to search for valid NAL start codes
- If buffer doesn't start with 0x00 0x00, search for first NAL start
- Skip garbage data before first valid NAL unit
- Return full buffer length when no NAL found (clears the buffer)
- Change forbidden_zero_bit error from fatal to skip-and-continue
Tested with 6 Twitch HLS sample files - all now process correctly.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Old versions of ccextractor accepted single-dash long options like
-quiet, -stdout, -autoprogram. The new Rust-based argument parser
(clap) only accepts double-dash options (--quiet, --stdout, etc.).
When users ran scripts with -quiet, clap parsed it as individual
short options -q -u -i -e -t and failed with exit code 7. Users
with stderr redirected never saw the error, causing silent failures
with zero-length output files.
This adds a normalize_legacy_option() function that pre-processes
arguments before passing them to clap:
- Single-dash long options (e.g., -quiet) convert to --quiet
- Double-dash options remain unchanged
- Short options like -o remain unchanged
- Numeric options like -1, -12 remain unchanged
Includes 6 unit tests for the new function.
Fixes#1576🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fix raw caption file processing that would stop at exactly 9:43:00 (2MB).
Root causes and fixes:
1. Premature EOF: After processing first chunk (BUFSIZE ~2MB), data->len
was never reset. On next iteration, general_get_more_data() calculated
want = BUFSIZE - len = 0 and returned EOF immediately.
Fix: Reset data->len = 0 after each chunk and change loop condition.
2. 32-bit integer overflow: The calculation cb_field1 * 1001 / 30 * 90
overflowed for large cb_field1 values (>1M). For example,
34,989,487 * 90 = 3,149,053,830 exceeds 32-bit signed max.
Fix: Cast cb_field1 to LLONG before multiplication.
3. Timing initialization: Raw mode needs min_pts=0, sync_pts=0, and
pts_set=MinPtsSet for correct fts_now calculation.
Tested with sample files from issue #1565:
- DTV3.raw: Now processes to 17:59:56 (was stopping at 9:43)
- DTV4.raw: Now processes to 14:00:00 (was stopping at 9:43)
- DTV5.raw: Now processes to 13:19:59 (was stopping at 9:43)
Closes#1565🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add a new --list-tracks (-L) option that lists all tracks found in
media files without processing them. This is useful for exploring
media files before caption extraction.
Supports:
- Matroska (MKV/WebM) files
- MP4/MOV files
- MPEG Transport Stream files
The feature is implemented entirely in Rust with native parsers for
each format, avoiding dependency on external libraries.
Closes#1669🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The previous WTV timing fix (commit 300f8ca6) set min_pts and pts_set=2
(MinPtsSet) but didn't set sync_pts. This caused the Rust timing code
to detect a massive PTS jump when processing WTV files with large
initial timestamps (e.g., files recorded at 18:38:23).
The PTS jump detection computes (current_pts - sync_pts), and with
sync_pts=0 but current_pts=6039323550 (18:38:23 in PTS units), the
difference exceeded MAX_DIF and triggered the jump handling, resulting
in empty output.
This fix sets sync_pts to the same value as min_pts when first
initializing timing, preventing the false PTS jump detection.
Test results:
- Before: WTV files with large initial PTS produced empty output
- After: Timestamps match expected ground truth exactly
(e.g., 00:00:00,601 --> 00:00:02,801 for first caption)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When using --udp or --tcp options, ccxr_demuxer_open() was called with
a NULL file pointer, causing a crash in CStr::from_ptr().
The fix checks if the file pointer is NULL before dereferencing it,
and uses an empty string for network input modes.
Fixes#1846🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>