* ci: add Linux and macOS workflows for FFmpeg MP4 build
Adds a cmake_ffmpeg_mp4 job to both build_linux.yml and build_mac.yml
that configures CCExtractor with -DWITH_FFMPEG=ON -DWITH_OCR=ON
-DWITH_HARDSUBX=ON and builds it, so the FFmpeg-based MP4 demuxer
path is exercised on every PR. Linux job pulls the full set of
ffmpeg/tesseract/leptonica dev packages (including libavdevice-dev);
macOS job installs the corresponding Homebrew bottles.
* build: wire FFmpeg libs into ccx_rust link order
Add the compile-time option to build CCExtractor's MP4 demuxer on top
of libavformat: set -DENABLE_FFMPEG_MP4 and pull libswresample as a
required dependency alongside libavformat/libavutil/libavcodec/
libavfilter/libswscale when -DWITH_FFMPEG=ON.
Link-order handling. Corrosion places ccx_rust at the end of the
ccextractor link line. On Linux (GNU ld), ccx_rust contains rsmpeg
which references FFmpeg symbols like swr_get_out_samples, so the
FFmpeg shared libs must appear after ccx_rust. Collect them into a
separate EXTRA_FFMPEG_LIBS variable and attach them as
INTERFACE_LINK_LIBRARIES on the ccx_rust target so CMake emits them
right after ccx_rust. Make ccx's own link dependencies PRIVATE so
the same libs don't propagate earlier and get deduplicated against
the INTERFACE copy.
Force bridge symbols to be pulled from libccx. GNU ld only pulls
object files from a static archive when they resolve currently-needed
symbols. Bridge functions in libccx.a (ccx_mp4_process_nal_sample,
ccx_mp4_process_cc_packet, etc.) aren't needed until libccx_rust.a
is processed, but libccx.a precedes it on the command line. Use
-Wl,--undefined=<symbol> on ccextractor for each bridge entry
point so the linker pulls them early — same pattern already used
for decode_vbi/do_cb/store_hdcc.
Rust crate wiring. Add the optional rsmpeg dependency guarded behind
the enable_mp4_ffmpeg feature, with platform-specific feature flags
(link_system_ffmpeg on Linux + macOS, link_vcpkg_ffmpeg on Windows,
all using the ffmpeg7 bindings). Extend bindgen's wrapper.h to
expose the bridge headers so Rust can call back into ccx from
mp4_rust_bridge.c.
* feat(mp4): FFmpeg MP4 demuxer with GPAC-parity caption extraction
Alternative MP4 demuxing backend built on rsmpeg (Rust FFmpeg bindings)
that matches GPAC's caption output on every sample reviewed. Activated
via -DWITH_FFMPEG=ON at compile time; default build keeps the GPAC
path untouched.
Architecture
- src/rust/src/demuxer/mp4.rs: opens the MP4 with rsmpeg, classifies
tracks (AVC, HEVC, c608, c708, tx3g), and drives packet dispatch.
- src/rust/src/mp4_ffmpeg_exports.rs: #[no_mangle] entry points
(ccxr_processmp4, ccxr_dumpchapters) called from ccextractor.c.
- src/lib_ccx/mp4_rust_bridge.c/.h: thin C shim around do_NAL,
process608, process_cc_data, ccdp_find_data, store_hdcc, and
encode_sub so the Rust side can feed decoded payloads into
CCExtractor's existing CEA-608/708 pipeline.
Caption parity
GPAC-equivalent output on all six samples from the Apr 18 review:
- 132d7df7e993.mov 108 290 B byte-identical
- 1974a299f050.mov 127 828 B byte-identical
- 99e5eaafdc55.mov 164 099 B byte-identical
- 8849331ddae9.mp4 48 485 B identical size, content, caption
count; uniform ~2 ms timing shift
- b2771c84c2a3.mp4 2 607 B byte-identical
- 5df914ce773d.mp4 1 164 B byte-identical
SEI-embedded CEA-608 in H.264 video
The last caption finishing on the final sample was never encoded
because the interleaved av_read_frame loop exited without an
equivalent of GPAC's per-track encode_sub. Drain sub.got_output at
EOF. Intentionally avoid calling process_hdcc there: the last IDR's
slice_header already flushed the HD-CC buffer, and re-running
process_hdcc re-emits partial post-IDR caption state as trailing
garbage.
c608 payload handling
libavformat delivers c608/c708 samples in two shapes:
1) atom-wrapped raw 608 pairs — [u32 length][4cc cdat|cdt2|ccdp]
[payload]. Strip the 8-byte header to match GPAC's process_clcp.
2) bare cc_data triplets — [cc_info][b1][b2]. Detect via len % 3 == 0
and (payload[0] & 0xF8) == 0xF8. For c608 tracks, extract each
field-1/field-2 pair from the triplet, set dec_ctx->current_field
so process608 picks the right decoder context, and call
process608 directly. Routing through do_cb would hit its
CCX_H264 guard (set by interleaved H.264 packets) and suppress
the cb_field increments process608 relies on for
caption-boundary timing, which merged short captions into their
successors. For c708, keep ccdp_find_data + process_cc_data.
Bridge design
Single unified entry point ccx_mp4_process_nal_sample(..., int
is_hevc, ...) replaces the separate AVC/HEVC helpers — their NAL
iteration was ~90% identical. Uses utility.h's RB16/RB32 macros
instead of hand-rolled byte-swap helpers. Handles HEVC's
end-of-sample cc_data flush inline.
Supported and unsupported tracks are documented in the mp4.rs
module header. Known limitation: dvdsub/bitmap subtitles in MP4
are not decoded (neither GPAC nor this backend handles them).
* chore(rust): fix clippy 1.95 warnings blocking CI
The format_rust CI job runs `cargo clippy --lib -- -D warnings` and
rust-1.95.0 promoted a handful of lints that hadn't been tripped on
master before this branch went through CI. Address all six:
collapsible_match (4 sites):
- src/demuxer/demux.rs: fold the nested `if matches!(...)` inside
the `Some(false) =>` arm into a match guard on the arm itself.
- src/encoder/common.rs: three header-write arms (Ccd, Scc, Raw)
each had a nested `if write_raw(...) == -1 { return -1; }`.
Collapse each into a match guard; the body now just logs and
returns.
unnecessary_cast (2 sites):
- src/libccxr_exports/demuxer.rs: drop `as i64` from
demux_ctx.get_filesize() — it already returns i64.
- src/parser.rs: drop `as u64` from `t as u64` when writing to
UTC_REFVALUE — t is already u64.
Also two lints in the new MP4 demuxer file from this branch:
- if_same_then_else at src/demuxer/mp4.rs:198 — the FOURCC_TX3G
and AV_CODEC_ID_MOV_TEXT arms both produced TrackType::Tx3g;
same for FOURCC_C608 and AV_CODEC_ID_EIA_608 → TrackType::Cea608.
Fold each pair into a single `||` branch.
- manual_is_multiple_of at src/demuxer/mp4.rs:399 —
`packet_count % 100 == 0` → `packet_count.is_multiple_of(100)`.
No behavior change. Caption parity against GPAC on all six mentor
samples verified post-fix.
---------
Co-authored-by: GAURAV KARMAKAR <gaurav02081@users.noreply.github.com>
* fix: flush pending EIT sections in EPG_free() before freeing buffers
* ci: add dual build artifacts to compare C vs Rust code paths
Add -min-rust flag to linux/build that passes -DDISABLE_RUST to gcc,
causing switchable modules (DTVCC, demuxer, AVC, networking, hex utils)
to use their C implementations instead of Rust. The Rust library still
compiles since many modules are Rust-only.
The Linux CI now produces two artifacts:
- "CCExtractor Linux build" — min Rust (C paths where available)
- "CCExtractor Linux build (with migrations)" — max Rust
Both should produce identical output on the sample platform. If they
diverge, it means a Rust port introduced a behavioral difference.
The sample platform will need a corresponding update to recognize and
test the new "with migrations" artifact.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Varadraj75 <agrawalvaradraj2007@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Improve CI caching and fix Linux build dependencies
- release.yml: Replace manual vcpkg clone + binary cache with
lukka/run-vcpkg@v11 and installed-packages caching (matching
build_windows.yml), add Cargo registry caching
- build_linux.yml: Add FFmpeg dev packages to build_autoconf, cmake,
and build_rust jobs — libgpac on newer Ubuntu requires FFmpeg 6.x
libraries at link time
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix WinGet publish: add fork-user to point to CCExtractor org fork
The winget-releaser action defaults to creating branches on the token
owner's fork (cfsmp3/winget-pkgs), which doesn't exist. The actual
fork is at CCExtractor/winget-pkgs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The CCExtractor/sample-platform test runner does exact string matching
on artifact names to find builds for testing. If these names are changed
without updating Artifact_names in sample-platform, CI tests silently
stop running.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Build workflows were not triggering on CMakeLists.txt changes.
Added **CMakeLists.txt and **.cmake patterns to path filters for:
- build_linux.yml
- build_mac.yml
- build_windows.yml
- build_docker.yml
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: unpack gpac
* fix: linux ci
* fix: mac build
* fix: remove unused [no ci]
* fix: ignore config.h [no ci]
* temp commit, will drop this soon
* fix: install gpac
* fix: gpac
* fix: formatting
* fix: preproccessor directive
* fix: comment display version for now
* fix: display dlls code
* fix: bundle vcruntime in hardsubx windows
* fix: again
* fix: erros in ci
* fix: ci
* fix: add vcruntime in additional dependencies
* fix: try to copy vcruntime after build
* fix: space in runtime library
* fix: remove for now [no ci]
* fix: things in vcxproj
* fix: ci for leptonica sys
* fix: docs
* fix: copy dlls on post build event
* fix: copy vcruntime after build
* fix: bump leptonica-sys to 0.4.3 and update Cargo.lock
* fix: bump rust version to 1.57.0 and build vcpkg for window hardsubx builds
* fix: add Bcrypt dependency
* fix: switch to rust stable
* chore: bump package versions
* fix: try to remove i686 to fix error
* fix: install tesseract and lint fixes
* fix: try using ffmpeg the third
* fix: include headers
* fix: add rsmpeg
* fix: switch default triplet to static md
* fix: import errors
* fix: directory path
* fix: pre build commands
* fix: update vcxproj
* fix: linux ci
* fix: ci fixes
* chore: lint fixes
* fix: error
* fix: copy include files
* fix: ci error
* fix: link swresample lib
* fix: some errors
* fix: include directory path and include all libraries
* fix: try to add library directories
* fix: fixes in libraries
* fix: formatting ci
* fix: mflat errors
* fix: libcurl
* fix: preprocessor definitions
* fix: add libcrypto
* fix: remove lib_hash to fix conflicts (we have libcrypto already)
* fix: add avcodec and avformat dependencies on windows
* fix: add remaining deps that may fix the build
* fix: add crypt depdency
* fix: rename conflicting names
* Revert "fix: remove lib_hash to fix conflicts (we have libcrypto already)"
This reverts commit f57ff716ed.
* fix: prefix with CC_
* fix: post build actions
* fix: ocr error
* Revert "fix: ocr error"
This reverts commit 92599454b6.
* fix: xcopy error
* fix: generated file name for x64
* fix: ocr error
* fix: add item group at top to see if it works
* fix: remove unwanted headers, removed \\ from VCPKG_ROOT, remove unwanted includes in vcxproj
* fix: add libpng for non hardsubx, comment the broken ocr code again
* fix: libpng path
* feat: add lib png headers in ClCompile
* fix: png.h not found
* fix: last try for ocr fix
* fix: libpng not found
* fix: cl compile headers
* fix: libpng and ocr
* fix: libpng error
* fix: redefinition error
* fix: zlib for non hardsubx
* fix: lib names
* fix: zlib.h not found
* set up bindings conversion of hardsubx utility functions (and structs) and set up the module
* add low level ffmpeg rust binding
* Methods ported:
- convert_pts_to_ns
- convert_pts_to_ms
- convert_pts_to_s
A pure rust method was added called _edit_distance_rec that implements levenstein distance calculation using recursion and dynamic programming
The port of edit_distance_rec is simply a wrapper that calls above function.
This redundancy won't be nevessary as more downstream modules are ported to Rust
* put C code of hardsubx_utility under define rust flag
* run formatter
* make compilation of hardsubx rust modules conditional on the HARDSUBX and the OCR flags. Make ffmpeg a conditional dependency based on those flags
* remove namespaced dependency in cargo because that is a nightly feature
* add conditioal compilatio of ffmpeg related bindigs in build.rs
* make clang argument of -DENABLE_HARDSUBX conditional on cargo feature of hardsubx_ocr
* enable specific relevant features for ffmpeg-sys-next
* enable hardsubx_ocr feature in windows build
* add build feature in ffmpeg-sys-next
* ffmpeg build feature is conditional on platform
* Revert "ffmpeg build feature is conditional on platform"
This reverts commit e456fee942.
This is because conditional features do not work in cargo toml
* install yasm in the linux build github action for ocr and hardsubx enabled cmake
* turn globals to locals to reduce code
* remove redundant attributes
* style changes
* make import of ffmpeg-sys-next conditional on hardsubx_ocr flag
* add --all-features flag in clippy for github workflow
* run formatter
* fix clippy command
* install yasm as part of rust format build check
* install libtesseract-dev etc. for clippy build test
* readability change
* declare the function edit_distance as unsafe
* remove commented code
* formatting changes
* combine declaration and assignment
* add build command for building hardsubx rust
context to issue: #1445
* make hardsubx rust work with autoconf build. For issue: #1445
* update autoconf for mac for issue #1445
* Use rust by default and add -WITHOUT_RUST flag
* Fix for shell and autoconf builds
* change directory for version check
* change to staticlib
* Update windows to build rust
* fix formatting
* add information about 708 decoder in version flag
* revert file mode to 644
* Use x86 for OCR releases
* fix flushing bug
* fix formatting
* update lib names
* remove bazel
* update changelog
