starred/sharpcompress
1
0
Fork 0
mirror of https://github.com/adamhathcock/sharpcompress.git synced 2026-02-06 21:26:07 +00:00
Code Issues 975 Packages Projects Releases 20 Wiki Activity

LzmaStream does not free memory #335

New Issue
Open
opened 2026-01-29 22:10:20 +00:00 by claunia · 5 comments
claunia commented 2026-01-29 22:10:20 +00:00
Owner
Copy Link

Originally created by @claunia on GitHub (Dec 1, 2018).

When you create a new LzmaStream it uses memory of about dictionary*11.5.

You expect that memory to go away when you do LzmaStream.Close() or LzmaStream.Dispose().

However that's not the case, and under certain conditions the .NET memory allocator doesn't free the object until the method that created it has exited, meaning that a single method can easily run out of process memory with a handful of LzmaStream initializations, that are not cleared even when manually calling the Garbage Collector.

The problem was discovered in https://github.com/claunia/DiscImageChef/issues/210

Originally created by @claunia on GitHub (Dec 1, 2018). When you create a new LzmaStream it uses memory of about dictionary*11.5. You expect that memory to go away when you do LzmaStream.Close() or LzmaStream.Dispose(). However that's not the case, and under certain conditions the .NET memory allocator doesn't free the object until the method that created it has exited, meaning that a single method can easily run out of process memory with a handful of LzmaStream initializations, that are not cleared even when manually calling the Garbage Collector. The problem was discovered in https://github.com/claunia/DiscImageChef/issues/210
claunia commented 2026-01-29 22:10:21 +00:00
Author
Owner
Copy Link

@adamhathcock commented on GitHub (Dec 3, 2018):

What are you saying? You're disposing the LzmaStream but still holding on to that instance? The memory it allocates would be freed once you don't have any pointers to the LzmaStream. I don't believe it uses memory in such a way that it leaks it if the LzmaStream itself is garbage collected.

@adamhathcock commented on GitHub (Dec 3, 2018): What are you saying? You're disposing the LzmaStream but still holding on to that instance? The memory it allocates would be freed once you don't have any pointers to the LzmaStream. I don't believe it uses memory in such a way that it leaks it if the LzmaStream itself is garbage collected.
claunia commented 2026-01-29 22:10:21 +00:00
Author
Owner
Copy Link

@claunia commented on GitHub (Dec 31, 2018):

Hi @adamhathcock sorry for late response.

As I explained what I observed is that .NET is not freeing memory in-method.

Should be pretty easy to replicate.

Just create something like this:

void CrashMethod()
{
    for(int i = 0; i < 10; i++)
    {
        var stream = new LzmaStream();
        // Write to it
        stream.Close(); // Same if you do stream = null;
    }
}

Then if you run it thru the profiler you will see that the memory is never released until CrashMethod() has exited.

It also exhibits this behaviour:

Stream stream;

void FatherMethod()
{
    stream = new LzmaStream();
    // Write to it, allocates memory
    stream.Close(); // Memory is NOT freed
    ChildMethod();
}

void ChildMethod()
{
    for(int i = 0; i < 10; i++)
    {
        // The first run of the loop will free the memory from FatherMethod(), but not the one allocated on the rest of the loop
        var stream = new LzmaStream();
        // Write to it
        stream.Close(); // Same if you do stream = null;
    }
}

And in the end the solution I found was:

void DoesNotCrash()
{
    Stream destination; // Where the LzmaStream will really write the data
    for(int i = 0; i < 10; i++)
    {
        destination = new FileStream("foobar");
        DoCompress(destination);
        destination.Close();
    }
}

void DoCompress(Stream destination)
{
    var stream = new LzmaStream(destination);
    // write data, memory is allocated here
} // Not even need to call Close(). Memory gets freed here

So the end result is that the .NET Framework does not free any kind of memory usage that has been allocated until the method that allocates such memory exits. Calling the garbage collector makes no difference in any of its variants.

I'm not particularly sure if it can be solved inside LzmaStream (maybe clearing the dictionary array on close?) but would be good to be documented, as I could not find documentation of this particular framework behaviour anywhere, having to discover it using non-free memory profiling tools.

@claunia commented on GitHub (Dec 31, 2018): Hi @adamhathcock sorry for late response. As I explained what I observed is that .NET is not freeing memory in-method. Should be pretty easy to replicate. Just create something like this: ```cs void CrashMethod() { for(int i = 0; i < 10; i++) { var stream = new LzmaStream(); // Write to it stream.Close(); // Same if you do stream = null; } } ``` Then if you run it thru the profiler you will see that the memory is never released until `CrashMethod()` has exited. It also exhibits this behaviour: ```cs Stream stream; void FatherMethod() { stream = new LzmaStream(); // Write to it, allocates memory stream.Close(); // Memory is NOT freed ChildMethod(); } void ChildMethod() { for(int i = 0; i < 10; i++) { // The first run of the loop will free the memory from FatherMethod(), but not the one allocated on the rest of the loop var stream = new LzmaStream(); // Write to it stream.Close(); // Same if you do stream = null; } } ``` And in the end the solution I found was: ```cs void DoesNotCrash() { Stream destination; // Where the LzmaStream will really write the data for(int i = 0; i < 10; i++) { destination = new FileStream("foobar"); DoCompress(destination); destination.Close(); } } void DoCompress(Stream destination) { var stream = new LzmaStream(destination); // write data, memory is allocated here } // Not even need to call Close(). Memory gets freed here ``` So the end result is that the .NET Framework does not free any kind of memory usage that has been allocated until the method that allocates such memory exits. Calling the garbage collector makes no difference in any of its variants. I'm not particularly sure if it can be solved inside LzmaStream (maybe clearing the dictionary array on close?) but would be good to be documented, as I could not find documentation of this particular framework behaviour anywhere, having to discover it using non-free memory profiling tools.
claunia commented 2026-01-29 22:10:22 +00:00
Author
Owner
Copy Link

@claunia commented on GitHub (Dec 31, 2018):

P.S.: happy new year

@claunia commented on GitHub (Dec 31, 2018): P.S.: happy new year
claunia commented 2026-01-29 22:10:23 +00:00
Author
Owner
Copy Link

@adamhathcock commented on GitHub (Jan 3, 2019):

There aren’t any unmanaged resources to consider not any statics so what’s happening is all up to the GC.

I could be mistaken but the memory allocation is necessary in the constructor and isn’t reused. An arraypool could possibly be used but Dispose would need to be explicitly used in any case.

@adamhathcock commented on GitHub (Jan 3, 2019): There aren’t any unmanaged resources to consider not any statics so what’s happening is all up to the GC. I could be mistaken but the memory allocation is necessary in the constructor and isn’t reused. An arraypool could possibly be used but Dispose would need to be explicitly used in any case.
claunia commented 2026-01-29 22:10:23 +00:00
Author
Owner
Copy Link

@claunia commented on GitHub (Jan 3, 2019):

If that can help...
If not at least documenting the behaviour, so people know what to except when using the knowingly memory intensive LZMA algorithm.

@claunia commented on GitHub (Jan 3, 2019): If that can help... If not at least documenting the behaviour, so people know what to except when using the knowingly memory intensive LZMA algorithm.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
duplicate
enhancement
feedback requested
not bug
pull-request
Mirrored from GitHub Pull Request
 question
up for grabs
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/sharpcompress#335
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?