mirror of
https://github.com/CCExtractor/ccextractor.git
synced 2026-07-08 18:06:30 +00:00
e4443a74ba
* ci: add Linux and macOS workflows for FFmpeg MP4 build
Adds a cmake_ffmpeg_mp4 job to both build_linux.yml and build_mac.yml
that configures CCExtractor with -DWITH_FFMPEG=ON -DWITH_OCR=ON
-DWITH_HARDSUBX=ON and builds it, so the FFmpeg-based MP4 demuxer
path is exercised on every PR. Linux job pulls the full set of
ffmpeg/tesseract/leptonica dev packages (including libavdevice-dev);
macOS job installs the corresponding Homebrew bottles.
* build: wire FFmpeg libs into ccx_rust link order
Add the compile-time option to build CCExtractor's MP4 demuxer on top
of libavformat: set -DENABLE_FFMPEG_MP4 and pull libswresample as a
required dependency alongside libavformat/libavutil/libavcodec/
libavfilter/libswscale when -DWITH_FFMPEG=ON.
Link-order handling. Corrosion places ccx_rust at the end of the
ccextractor link line. On Linux (GNU ld), ccx_rust contains rsmpeg
which references FFmpeg symbols like swr_get_out_samples, so the
FFmpeg shared libs must appear after ccx_rust. Collect them into a
separate EXTRA_FFMPEG_LIBS variable and attach them as
INTERFACE_LINK_LIBRARIES on the ccx_rust target so CMake emits them
right after ccx_rust. Make ccx's own link dependencies PRIVATE so
the same libs don't propagate earlier and get deduplicated against
the INTERFACE copy.
Force bridge symbols to be pulled from libccx. GNU ld only pulls
object files from a static archive when they resolve currently-needed
symbols. Bridge functions in libccx.a (ccx_mp4_process_nal_sample,
ccx_mp4_process_cc_packet, etc.) aren't needed until libccx_rust.a
is processed, but libccx.a precedes it on the command line. Use
-Wl,--undefined=<symbol> on ccextractor for each bridge entry
point so the linker pulls them early — same pattern already used
for decode_vbi/do_cb/store_hdcc.
Rust crate wiring. Add the optional rsmpeg dependency guarded behind
the enable_mp4_ffmpeg feature, with platform-specific feature flags
(link_system_ffmpeg on Linux + macOS, link_vcpkg_ffmpeg on Windows,
all using the ffmpeg7 bindings). Extend bindgen's wrapper.h to
expose the bridge headers so Rust can call back into ccx from
mp4_rust_bridge.c.
* feat(mp4): FFmpeg MP4 demuxer with GPAC-parity caption extraction
Alternative MP4 demuxing backend built on rsmpeg (Rust FFmpeg bindings)
that matches GPAC's caption output on every sample reviewed. Activated
via -DWITH_FFMPEG=ON at compile time; default build keeps the GPAC
path untouched.
Architecture
- src/rust/src/demuxer/mp4.rs: opens the MP4 with rsmpeg, classifies
tracks (AVC, HEVC, c608, c708, tx3g), and drives packet dispatch.
- src/rust/src/mp4_ffmpeg_exports.rs: #[no_mangle] entry points
(ccxr_processmp4, ccxr_dumpchapters) called from ccextractor.c.
- src/lib_ccx/mp4_rust_bridge.c/.h: thin C shim around do_NAL,
process608, process_cc_data, ccdp_find_data, store_hdcc, and
encode_sub so the Rust side can feed decoded payloads into
CCExtractor's existing CEA-608/708 pipeline.
Caption parity
GPAC-equivalent output on all six samples from the Apr 18 review:
- 132d7df7e993.mov 108 290 B byte-identical
- 1974a299f050.mov 127 828 B byte-identical
- 99e5eaafdc55.mov 164 099 B byte-identical
- 8849331ddae9.mp4 48 485 B identical size, content, caption
count; uniform ~2 ms timing shift
- b2771c84c2a3.mp4 2 607 B byte-identical
- 5df914ce773d.mp4 1 164 B byte-identical
SEI-embedded CEA-608 in H.264 video
The last caption finishing on the final sample was never encoded
because the interleaved av_read_frame loop exited without an
equivalent of GPAC's per-track encode_sub. Drain sub.got_output at
EOF. Intentionally avoid calling process_hdcc there: the last IDR's
slice_header already flushed the HD-CC buffer, and re-running
process_hdcc re-emits partial post-IDR caption state as trailing
garbage.
c608 payload handling
libavformat delivers c608/c708 samples in two shapes:
1) atom-wrapped raw 608 pairs — [u32 length][4cc cdat|cdt2|ccdp]
[payload]. Strip the 8-byte header to match GPAC's process_clcp.
2) bare cc_data triplets — [cc_info][b1][b2]. Detect via len % 3 == 0
and (payload[0] & 0xF8) == 0xF8. For c608 tracks, extract each
field-1/field-2 pair from the triplet, set dec_ctx->current_field
so process608 picks the right decoder context, and call
process608 directly. Routing through do_cb would hit its
CCX_H264 guard (set by interleaved H.264 packets) and suppress
the cb_field increments process608 relies on for
caption-boundary timing, which merged short captions into their
successors. For c708, keep ccdp_find_data + process_cc_data.
Bridge design
Single unified entry point ccx_mp4_process_nal_sample(..., int
is_hevc, ...) replaces the separate AVC/HEVC helpers — their NAL
iteration was ~90% identical. Uses utility.h's RB16/RB32 macros
instead of hand-rolled byte-swap helpers. Handles HEVC's
end-of-sample cc_data flush inline.
Supported and unsupported tracks are documented in the mp4.rs
module header. Known limitation: dvdsub/bitmap subtitles in MP4
are not decoded (neither GPAC nor this backend handles them).
* chore(rust): fix clippy 1.95 warnings blocking CI
The format_rust CI job runs `cargo clippy --lib -- -D warnings` and
rust-1.95.0 promoted a handful of lints that hadn't been tripped on
master before this branch went through CI. Address all six:
collapsible_match (4 sites):
- src/demuxer/demux.rs: fold the nested `if matches!(...)` inside
the `Some(false) =>` arm into a match guard on the arm itself.
- src/encoder/common.rs: three header-write arms (Ccd, Scc, Raw)
each had a nested `if write_raw(...) == -1 { return -1; }`.
Collapse each into a match guard; the body now just logs and
returns.
unnecessary_cast (2 sites):
- src/libccxr_exports/demuxer.rs: drop `as i64` from
demux_ctx.get_filesize() — it already returns i64.
- src/parser.rs: drop `as u64` from `t as u64` when writing to
UTC_REFVALUE — t is already u64.
Also two lints in the new MP4 demuxer file from this branch:
- if_same_then_else at src/demuxer/mp4.rs:198 — the FOURCC_TX3G
and AV_CODEC_ID_MOV_TEXT arms both produced TrackType::Tx3g;
same for FOURCC_C608 and AV_CODEC_ID_EIA_608 → TrackType::Cea608.
Fold each pair into a single `||` branch.
- manual_is_multiple_of at src/demuxer/mp4.rs:399 —
`packet_count % 100 == 0` → `packet_count.is_multiple_of(100)`.
No behavior change. Caption parity against GPAC on all six mentor
samples verified post-fix.
---------
Co-authored-by: GAURAV KARMAKAR <gaurav02081@users.noreply.github.com>
164 lines
5.7 KiB
YAML
164 lines
5.7 KiB
YAML
name: Build CCExtractor on Linux
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
push:
|
|
paths:
|
|
- '.github/workflows/build_linux.yml'
|
|
- '**.c'
|
|
- '**.h'
|
|
- '**CMakeLists.txt'
|
|
- '**.cmake'
|
|
- '**Makefile**'
|
|
- 'linux/**'
|
|
- 'package_creators/**'
|
|
- 'src/rust/**'
|
|
pull_request:
|
|
types: [opened, synchronize, reopened]
|
|
paths:
|
|
- '.github/workflows/build_linux.yml'
|
|
- '**.c'
|
|
- '**.h'
|
|
- '**CMakeLists.txt'
|
|
- '**.cmake'
|
|
- '**Makefile**'
|
|
- 'linux/**'
|
|
- 'package_creators/**'
|
|
- 'src/rust/**'
|
|
jobs:
|
|
build_shell:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Install dependencies
|
|
run: sudo apt update && sudo apt-get install libgpac-dev libtesseract-dev libavcodec-dev libavdevice-dev libx11-dev libxcb1-dev libxcb-shm0-dev
|
|
- uses: actions/checkout@v6
|
|
- name: build
|
|
run: ./build -hardsubx -min-rust
|
|
working-directory: ./linux
|
|
- name: Display version information
|
|
run: ./ccextractor --version
|
|
working-directory: ./linux
|
|
- name: Prepare artifacts
|
|
run: mkdir ./linux/artifacts
|
|
- name: Copy release artifact
|
|
run: cp ./linux/ccextractor ./linux/artifacts/
|
|
# NOTE: The sample-platform test runner (CCExtractor/sample-platform)
|
|
# matches artifact names exactly. Update Artifact_names in
|
|
# mod_ci/controllers.py there if you rename this artifact.
|
|
- uses: actions/upload-artifact@v7
|
|
with:
|
|
name: CCExtractor Linux build
|
|
path: ./linux/artifacts
|
|
build_shell_migrations:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Install dependencies
|
|
run: sudo apt update && sudo apt-get install libgpac-dev libtesseract-dev libavcodec-dev libavdevice-dev libx11-dev libxcb1-dev libxcb-shm0-dev
|
|
- uses: actions/checkout@v6
|
|
- name: build
|
|
run: ./build -hardsubx
|
|
working-directory: ./linux
|
|
- name: Display version information
|
|
run: ./ccextractor --version
|
|
working-directory: ./linux
|
|
- name: Prepare artifacts
|
|
run: mkdir ./linux/artifacts
|
|
- name: Copy release artifact
|
|
run: cp ./linux/ccextractor ./linux/artifacts/
|
|
# NOTE: The sample-platform test runner (CCExtractor/sample-platform)
|
|
# matches artifact names exactly. Update Artifact_names in
|
|
# mod_ci/controllers.py there if you rename this artifact.
|
|
- uses: actions/upload-artifact@v7
|
|
with:
|
|
name: CCExtractor Linux build (with migrations)
|
|
path: ./linux/artifacts
|
|
build_autoconf:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Install dependencies
|
|
run: sudo apt update && sudo apt-get install libgpac-dev libavcodec-dev libavformat-dev libavutil-dev libswresample-dev libswscale-dev
|
|
- uses: actions/checkout@v6
|
|
- name: run autogen
|
|
run: ./autogen.sh
|
|
working-directory: ./linux
|
|
- name: configure
|
|
run: ./configure --enable-debug
|
|
working-directory: ./linux
|
|
- name: make
|
|
run: make
|
|
working-directory: ./linux
|
|
- name: Display version information
|
|
run: ./ccextractor --version
|
|
working-directory: ./linux
|
|
cmake:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Install dependencies
|
|
run: sudo apt update && sudo apt-get install libgpac-dev libavcodec-dev libavformat-dev libavutil-dev libswresample-dev libswscale-dev
|
|
- uses: actions/checkout@v6
|
|
- name: cmake
|
|
run: mkdir build && cd build && cmake ../src
|
|
- name: build
|
|
run: make -j$(nproc)
|
|
working-directory: build
|
|
- name: Display version information
|
|
run: ./build/ccextractor --version
|
|
cmake_ffmpeg_mp4:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Install dependencies
|
|
run: sudo apt update && sudo apt-get install libgpac-dev libtesseract-dev libleptonica-dev libavcodec-dev libavformat-dev libavutil-dev libavfilter-dev libavdevice-dev libswresample-dev libswscale-dev
|
|
- uses: actions/checkout@v6
|
|
- name: cmake (default GPAC build)
|
|
run: mkdir build && cd build && cmake ../src
|
|
- name: build (default GPAC build)
|
|
run: make -j$(nproc)
|
|
working-directory: build
|
|
- name: Display version information (GPAC build)
|
|
run: ./build/ccextractor --version
|
|
- name: cmake (FFmpeg MP4 build)
|
|
run: mkdir build_ffmpeg && cd build_ffmpeg && cmake -DWITH_FFMPEG=ON -DWITH_OCR=ON -DWITH_HARDSUBX=ON ../src
|
|
- name: build (FFmpeg MP4 build)
|
|
run: make -j$(nproc)
|
|
working-directory: build_ffmpeg
|
|
- name: Display version information (FFmpeg MP4 build)
|
|
run: ./build_ffmpeg/ccextractor --version
|
|
cmake_ocr_hardsubx:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
- name: Install dependencies
|
|
run: sudo apt update && sudo apt install libgpac-dev libtesseract-dev libavformat-dev libavdevice-dev libswscale-dev yasm
|
|
- name: cmake
|
|
run: |
|
|
mkdir build && cd build
|
|
cmake -DWITH_OCR=ON -DWITH_HARDSUBX=ON ../src
|
|
- name: build
|
|
run: |
|
|
make -j$(nproc)
|
|
working-directory: build
|
|
- name: Display version information
|
|
run: ./build/ccextractor --version
|
|
build_rust:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Install dependencies
|
|
run: sudo apt update && sudo apt-get install libgpac-dev libavcodec-dev libavformat-dev libavutil-dev libswresample-dev libswscale-dev
|
|
- uses: actions/checkout@v6
|
|
- name: cache
|
|
uses: actions/cache@v5
|
|
with:
|
|
path: |
|
|
src/rust/.cargo/registry
|
|
src/rust/.cargo/git
|
|
src/rust/target
|
|
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
|
|
restore-keys: ${{ runner.os }}-cargo-
|
|
- uses: actions-rs/toolchain@v1
|
|
with:
|
|
toolchain: stable
|
|
override: true
|
|
- name: build
|
|
run: cargo build
|
|
working-directory: ./src/rust
|