Reusing/resetting a brotli.Writer instance #496

New Issue

claunia · 2026-01-29T20:44:51Z

claunia commented

2026-01-29 20:44:51 +00:00

Originally created by @ankon on GitHub (Feb 12, 2024).

We saw good performance improvements in our go program when using a sync.Pool to cache and reset gzip writers rather than creating them anew each time.

I was wondering whether the same would be true for the brotli writer, too, but that one doesn't seem to have a trivial "reset this" method. On the underlying C level it looks like an encoder state can be "created"/"destroyed", but there are functions to "initialize" and "cleanup" as well.

Before I try to hack this together: Do you think that pooling/reusing of the encoder state could have a measurable/visible performance impact?

Originally created by @ankon on GitHub (Feb 12, 2024). We saw good performance improvements in our go program when using a sync.Pool to cache and reset gzip writers rather than creating them anew each time. I was wondering whether the same would be true for the brotli writer, too, but that one doesn't seem to have a trivial "reset this" method. On the underlying C level it looks like an encoder state can be "created"/"destroyed", but there are functions to "initialize" and "cleanup" as well. Before I try to hack this together: Do you think that pooling/reusing of the encoder state could have a measurable/visible performance impact?

claunia commented

2026-01-29 20:44:51 +00:00

@eustas commented on GitHub (Feb 12, 2024):

Initialize and cleanup are sort-of internal. Could you point me to the changes regarding gzip, so I could investigate the source of performance improvements.
I could assume that performance is impacted by JNI roundtrips. If that is true, then we could think of something to reuse / refurbish instances.

@eustas commented on GitHub (Feb 12, 2024): Initialize and cleanup are sort-of internal. Could you point me to the changes regarding gzip, so I could investigate the source of performance improvements. I could assume that performance is impacted by JNI roundtrips. If that is true, then we could think of something to reuse / refurbish instances.

claunia commented

2026-01-29 20:44:52 +00:00

@ankon commented on GitHub (Feb 13, 2024):

So for gzip a lot of places1,2 suggest this approach (pseudo-golang here):

writerPool := &sync.Pool{
  New: func() any {
	writer, _ := gzip.NewWriterLevel(nil, g.Level)
	return writer
  },
}

// later when using it to wrap an existing output writer
var w io.Writer
// Grab a possibly pooled gzip writer
writer := writerPool.Get().(*gzip.Writer)
// Reset its structures and point it to now write to w
writer.Reset(w)

Doing some benchmarks (see [1] for an idea) also for us showed that the number of allocations went down quite a bit, and I think that might be due to internal caches and buffers getting reused.

See also gzip.Writer's Reset documentation

@ankon commented on GitHub (Feb 13, 2024): So for gzip a lot of places[1](https://developer20.com/using-sync-pool/),[2](https://github.com/caddyserver/caddy/tree/master/modules/caddyhttp/encode) suggest this approach (pseudo-golang here): ```go writerPool := &sync.Pool{ New: func() any { writer, _ := gzip.NewWriterLevel(nil, g.Level) return writer }, } // later when using it to wrap an existing output writer var w io.Writer // Grab a possibly pooled gzip writer writer := writerPool.Get().(*gzip.Writer) // Reset its structures and point it to now write to w writer.Reset(w) ``` Doing some benchmarks (see [1] for an idea) also for us showed that the number of allocations went down quite a bit, and I _think_ that might be due to internal caches and buffers getting reused. See also [gzip.Writer's `Reset` documentation](https://pkg.go.dev/compress/gzip#Writer.Reset)

claunia referenced this issue

2026-01-29 20:52:38 +00:00

[PR #496] [CLOSED] Haiku: force undefine INFINITY for GCC2 as it's not const #1135

claunia referenced this issue

2026-01-29 20:52:38 +00:00

[PR #497] [MERGED] Update encoder #1136

claunia referenced this issue

2026-01-29 20:52:39 +00:00

[PR #496] Haiku: force undefine INFINITY for GCC2 as it's not const #1138

claunia referenced this issue

2026-01-29 20:52:39 +00:00

[PR #497] Update encoder #1139