Feature request: Write processed bytes to file (GSoC qualification task) #386

Closed
opened 2026-01-29 16:42:33 +00:00 by claunia · 6 comments
Owner

Originally created by @cfsmp3 on GitHub (Feb 21, 2018).

Something that would be really useful to debug some of the issues that only appear when processing network streams (in which packet loss occurs):

Add a parameter that allows to specify an output file in which read bytes are written to, so it's easy later to reproduce the issues reading that file.

This is a relatively easy task because you only need to do some changes in the code that reads from the stream, which is in file_functions.c

Main gotcha is about seeks, particularly with negative values.

Originally created by @cfsmp3 on GitHub (Feb 21, 2018). Something that would be really useful to debug some of the issues that only appear when processing network streams (in which packet loss occurs): Add a parameter that allows to specify an output file in which read bytes are written to, so it's easy later to reproduce the issues reading that file. This is a relatively easy task because you only need to do some changes in the code that reads from the stream, which is in file_functions.c Main gotcha is about seeks, particularly with negative values.
Author
Owner

@cfsmp3 commented on GitHub (Feb 27, 2018):

Did this myself (lousy hack but worked for what I needed).

@cfsmp3 commented on GitHub (Feb 27, 2018): Did this myself (lousy hack but worked for what I needed).
Author
Owner

@cfsmp3 commented on GitHub (Mar 1, 2018):

GSoC students: By all means come up with a decent solution that works for all kind of files, uses parameters and so on. My hack is terrible and in no way intended to be the final implementation.

@cfsmp3 commented on GitHub (Mar 1, 2018): GSoC students: By all means come up with a decent solution that works for all kind of files, uses parameters and so on. My hack is terrible and in no way intended to be the final implementation.
Author
Owner

@gnathion commented on GitHub (Mar 1, 2018):

Hi @cfsmp3,
This feature seems really interesting and would be of great use when implemented. I think it would be apt to keep it open.
And, if it is possible, could you push your implementation of it onto another branch so that I can work on it from there-on? Or would you prefer me working on this from scratch?

Awaiting your reply and looking forward to contribute!

@gnathion commented on GitHub (Mar 1, 2018): Hi @cfsmp3, This feature seems really interesting and would be of great use when implemented. I think it would be apt to keep it open. And, if it is possible, could you push your implementation of it onto another branch so that I can work on it from there-on? Or would you prefer me working on this from scratch? Awaiting your reply and looking forward to contribute!
Author
Owner

@cfsmp3 commented on GitHub (Mar 1, 2018):

My "implementation" is already on the code, just between #ifdef.

It's a 6 line hack in ts_functions.c that for each TS packet opens the
output file, writes the packet, and closes it. Hardcoded path, too. So
well, it couldn't be lousier, but it allowed me to save a problematic
network stream I was dealing with.

But my no means should this code be used as a reference.

On Thu, Mar 1, 2018 at 5:20 AM, gnathion notifications@github.com wrote:

Hi @cfsmp3 https://github.com/cfsmp3,
This feature seems really interesting and would be of great use when
implemented. I think it would be apt to keep it open.
And, if it is possible, could you push your implementation of it onto
another branch so that I can work on it from there-on? Or would you prefer
me working on this from scratch?

Awaiting your reply and looking forward to contribute!


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/CCExtractor/ccextractor/issues/933#issuecomment-369589265,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFrJ2XesvEVgRgHVCfN4xKfLdN-n8934ks5tZ_WlgaJpZM4SOLHp
.

@cfsmp3 commented on GitHub (Mar 1, 2018): My "implementation" is already on the code, just between #ifdef. It's a 6 line hack in ts_functions.c that for each TS packet opens the output file, writes the packet, and closes it. Hardcoded path, too. So well, it couldn't be lousier, but it allowed me to save a problematic network stream I was dealing with. But my no means should this code be used as a reference. On Thu, Mar 1, 2018 at 5:20 AM, gnathion <notifications@github.com> wrote: > Hi @cfsmp3 <https://github.com/cfsmp3>, > This feature seems really interesting and would be of great use when > implemented. I think it would be apt to keep it open. > And, if it is possible, could you push your implementation of it onto > another branch so that I can work on it from there-on? Or would you prefer > me working on this from scratch? > > Awaiting your reply and looking forward to contribute! > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/CCExtractor/ccextractor/issues/933#issuecomment-369589265>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AFrJ2XesvEVgRgHVCfN4xKfLdN-n8934ks5tZ_WlgaJpZM4SOLHp> > . >
Author
Owner

@gnathion commented on GitHub (Mar 1, 2018):

I made changes to the current code to write the packets to a specified output file (You can find it on my fork here, please have a look at it.
I tested this with a local .TS file and it seemed to be working perfectly (without seeks).

This is just the basic outlook of this feature after I spent some time understanding the code-base. It would be very helpful if you could review the code and put things into place. I can work on managing seeks once this is ironed out. (I was reluctant to open a PR for such a small change, I will do it once there is sufficient headway.)

Thank You.

@gnathion commented on GitHub (Mar 1, 2018): I made changes to the current code to write the packets to a specified output file (You can find it on my fork [here](https://github.com/gnathion/ccextractor/commit/d919f526bd072b37e80b9b6293501119cbb3d15b), please have a look at it. I tested this with a local .TS file and it seemed to be working perfectly (without seeks). This is just the basic outlook of this feature after I spent some time understanding the code-base. It would be very helpful if you could review the code and put things into place. I can work on managing seeks once this is ironed out. (I was reluctant to open a PR for such a small change, I will do it once there is sufficient headway.) Thank You.
Author
Owner

@cfsmp3 commented on GitHub (Mar 2, 2018):

Problem with this code is that it sucks. Big time.

  1. Only works with TS
  2. Opens and closes the output file for each packet
  3. Write operation is in the middle of the decoder instead of in the
    read/seek functions

About your PR I don't understand this part:
if (opt->write_format==CCX_OF_RCWT

How is the write format related to this? We can be processing a network
stream, generating a .srt file (CCX_OF_SRT) and also making a copy of the
network stream in a file which is totally unrelated to the subtitle output
file which is what opt->write_format is about.

Anyway what I did was a hack, really. Quickest possible way to get what I
needed, but it's pure crap otherwise.

On Thu, Mar 1, 2018 at 12:30 PM, gnathion notifications@github.com wrote:

I made changes to the current code to write the packets to a specified
output file (You can find it on my fork here
https://github.com/gnathion/ccextractor/commit/d919f526bd072b37e80b9b6293501119cbb3d15b,
please have a look at it.

This is just the basic outlook of this feature after I spent some time
understanding the code-base. It would be very helpful if you could review
the code and put things into place. I can work on managing seeks once this
is ironed out. (I was reluctant to open a PR for such a small change, I
will do it once there is sufficient headway.)

Thank You.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/CCExtractor/ccextractor/issues/933#issuecomment-369721010,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFrJ2e4qtqq0wXIPC1Pqto8FnBiQEOLwks5taFpWgaJpZM4SOLHp
.

@cfsmp3 commented on GitHub (Mar 2, 2018): Problem with this code is that it sucks. Big time. 1) Only works with TS 2) Opens and closes the output file for each packet 3) Write operation is in the middle of the decoder instead of in the read/seek functions About your PR I don't understand this part: if (opt->write_format==CCX_OF_RCWT How is the write format related to this? We can be processing a network stream, generating a .srt file (CCX_OF_SRT) and also making a copy of the network stream in a file which is totally unrelated to the subtitle output file which is what opt->write_format is about. Anyway what I did was a hack, really. Quickest possible way to get what I needed, but it's pure crap otherwise. On Thu, Mar 1, 2018 at 12:30 PM, gnathion <notifications@github.com> wrote: > I made changes to the current code to write the packets to a specified > output file (You can find it on my fork here > <https://github.com/gnathion/ccextractor/commit/d919f526bd072b37e80b9b6293501119cbb3d15b>, > please have a look at it. > > This is just the basic outlook of this feature after I spent some time > understanding the code-base. It would be very helpful if you could review > the code and put things into place. I can work on managing seeks once this > is ironed out. (I was reluctant to open a PR for such a small change, I > will do it once there is sufficient headway.) > > Thank You. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/CCExtractor/ccextractor/issues/933#issuecomment-369721010>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AFrJ2e4qtqq0wXIPC1Pqto8FnBiQEOLwks5taFpWgaJpZM4SOLHp> > . >
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/ccextractor#386