High Memory Consumption - 180 MB #135

New Issue

claunia · 2026-01-29T16:36:08Z

claunia commented

2026-01-29 16:36:08 +00:00

Originally created by @josecasillas on GitHub (Apr 19, 2016).

Originally assigned to: @Abhinav95 on GitHub.

Shelling out ccextractor instances on Linux from an app to process parallel streams. Feeding low-bit-rate video via stdin, using stream mode, disabled input buffering, captions going into stdout. Seems each instance occupies a fixed amount of ~180 MB of RAM, severely limiting what we can accomplish on each box.

Is it possible to configure the size of internal buffers more modestly? Any insight into the 180 MB RAM footprint?

Thank you!
Jose

Originally created by @josecasillas on GitHub (Apr 19, 2016). Originally assigned to: @Abhinav95 on GitHub. Shelling out ccextractor instances on Linux from an app to process parallel streams. Feeding low-bit-rate video via stdin, using stream mode, disabled input buffering, captions going into stdout. Seems each instance occupies a fixed amount of ~180 MB of RAM, severely limiting what we can accomplish on each box. Is it possible to configure the size of internal buffers more modestly? Any insight into the 180 MB RAM footprint? Thank you! Jose

claunia closed this issue

2026-01-29 16:36:09 +00:00

claunia commented

2026-01-29 16:36:10 +00:00

@Abhinav95 commented on GitHub (May 24, 2016):

I can confirm this issue. Every instance of ccextractor, (both Windows and Linux), occupies an amount of memory that ranges around 160-200 MB, depending on the parameters passed.

@Abhinav95 commented on GitHub (May 24, 2016): I can confirm this issue. Every instance of ccextractor, (both Windows and Linux), occupies an amount of memory that ranges around 160-200 MB, depending on the parameters passed. ![mem1](https://cloud.githubusercontent.com/assets/7457692/15503747/0db6a0ae-21d9-11e6-9a98-aea5fadb7325.png) ![mem2](https://cloud.githubusercontent.com/assets/7457692/15503746/0db6126a-21d9-11e6-9792-f691098f7031.png) ![mem3](https://cloud.githubusercontent.com/assets/7457692/15503748/0dc7705a-21d9-11e6-9dde-6fa95e47a025.png)

claunia commented

2026-01-29 16:36:10 +00:00

@cfsmp3 commented on GitHub (May 24, 2016):

So where does all this memory go? :-)

On Tue, May 24, 2016 at 5:10 AM, Abhinav Shukla notifications@github.com
wrote:

I can confirm this issue. Every instance of ccextractor, (both Windows and
Linux), occupies an amount of memory that ranges around 160-200 MB,
depending on the parameters passed.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/CCExtractor/ccextractor/issues/353#issuecomment-221249395

@cfsmp3 commented on GitHub (May 24, 2016): So where does all this memory go? :-) On Tue, May 24, 2016 at 5:10 AM, Abhinav Shukla notifications@github.com wrote: > I can confirm this issue. Every instance of ccextractor, (both Windows and > Linux), occupies an amount of memory that ranges around 160-200 MB, > depending on the parameters passed. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly or view it on GitHub > https://github.com/CCExtractor/ccextractor/issues/353#issuecomment-221249395

claunia commented

2026-01-29 16:36:11 +00:00

@Abhinav95 commented on GitHub (May 25, 2016):

I am using valgrind to analyze memory allocation as the program runs.
The major problem is that 187477304 bytes (179.79 MB) of memory is allocated to ctx in the init_libraries() call at the beginning of the program from main.
sizeof(struct lib_ccx_ctx) returns 187477304.

UPDATE:-
struct EIT_program eit_programs[TS_PMT_MAP_SIZE+1] has a size of 185795585, which is the problem.
This happens because EPG data for 60 days (60_24_7) is allocated every time the program is run, which should not be the case and wastes around 180 MB memory.

#define EPG_MAX_EVENTS 60*24*7
struct EIT_program
{
    uint32_t array_len;
    struct EPG_event epg_events[EPG_MAX_EVENTS];
};
#endif

@Abhinav95 commented on GitHub (May 25, 2016): I am using valgrind to analyze memory allocation as the program runs. The major problem is that 187477304 bytes (179.79 MB) of memory is allocated to `ctx` in the `init_libraries()` call at the beginning of the program from `main`. `sizeof(struct lib_ccx_ctx)` returns 187477304. UPDATE:- `struct EIT_program eit_programs[TS_PMT_MAP_SIZE+1]` has a size of 185795585, which is the problem. This happens because EPG data for 60 days (60_24_7) is allocated every time the program is run, which should not be the case and wastes around 180 MB memory. ``` #define EPG_MAX_EVENTS 60*24*7 struct EIT_program { uint32_t array_len; struct EPG_event epg_events[EPG_MAX_EVENTS]; }; #endif ```

claunia commented

2026-01-29 16:36:11 +00:00

@Abhinav95 commented on GitHub (May 26, 2016):

#373 fixes this issue.

@josecasillas You should expect a memory usage of atmost 25 MB per instance from now on, except for when passing the '-xmltv' parameter which will give memory usage around 200 MB(which I don't think you are using because you described just caption output to stdout). Good luck for your application :)

@Abhinav95 commented on GitHub (May 26, 2016): #373 fixes this issue. @josecasillas You should expect a memory usage of atmost 25 MB per instance from now on, except for when passing the '-xmltv' parameter which will give memory usage around 200 MB(which I don't think you are using because you described just caption output to stdout). Good luck for your application :)

claunia commented

2026-01-29 16:36:12 +00:00

@josecasillas commented on GitHub (May 27, 2016):

I can't thank you enough. Great job on this, and fast turn around. Thank you my friend!

@josecasillas commented on GitHub (May 27, 2016): I can't thank you enough. Great job on this, and fast turn around. Thank you my friend!

claunia commented

2026-01-29 16:36:12 +00:00

@josecasillas commented on GitHub (May 27, 2016):

Tested today with the new build and I'm seeing < 5 MB per instance! Memory no longer the limiting factor. Thanks again!

@josecasillas commented on GitHub (May 27, 2016): Tested today with the new build and I'm seeing < 5 MB per instance! Memory no longer the limiting factor. Thanks again!

claunia commented

2026-01-29 16:36:12 +00:00

@cfsmp3 commented on GitHub (May 27, 2016):

Great job Abhinav !

On Fri, May 27, 2016 at 10:47 AM, josecasillas notifications@github.com
wrote:

Tested today with the new build and I'm seeing < 5 MB per instance! Memory
no longer the limiting factor. Thanks again!

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/CCExtractor/ccextractor/issues/353#issuecomment-222210395,
or mute the thread
https://github.com/notifications/unsubscribe/AFrJ2XhGSN_FAyjxa4kNriXKEQyeAa45ks5qFy4ngaJpZM4IK9jH
.

@cfsmp3 commented on GitHub (May 27, 2016): Great job Abhinav ! On Fri, May 27, 2016 at 10:47 AM, josecasillas notifications@github.com wrote: > Tested today with the new build and I'm seeing < 5 MB per instance! Memory > no longer the limiting factor. Thanks again! > > — > You are receiving this because you modified the open/close state. > Reply to this email directly, view it on GitHub > https://github.com/CCExtractor/ccextractor/issues/353#issuecomment-222210395, > or mute the thread > https://github.com/notifications/unsubscribe/AFrJ2XhGSN_FAyjxa4kNriXKEQyeAa45ks5qFy4ngaJpZM4IK9jH > .

claunia referenced this issue

2026-01-29 16:58:26 +00:00

[PR #135] [MERGED] Fix warnings and a memory leak in EPG support #1029

claunia referenced this issue

2026-01-29 16:58:29 +00:00

[PR #135] Fix warnings and a memory leak in EPG support #1034

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/ccextractor#135