Repository navigation
runtime/pprof,runtime: new goroutine leak profile #74609
Description
Proposal Details
This proposal involves a change to the garbage collector that allows it detect some types of goroutine leaks (a.k.a. partial deadlocks in the academic community).
The approach uses memory reachability as computed by the garbage collector to single out goroutines blocked over concurrency primitives that cannot be accessed by any (transitively) runnable goroutines. It is already described at length in this publication.
The approach has been validated experimentally in several contexts. It was able to discover goroutine leaks:
- caused by the misuse of synchronization primitives (channels,
syncprimitives, and mixed) in 69 microbenchmarks derived from real-world concurrency bugs, - in 3111 test suites within Uber, for libraries using concurrency features, with a rate of 180 to 357 syntactically distinct goroutine leaks when compared to uber-go/goleak (which is only applicable to unit tests)
- in a production service within Uber, where it was able to find 3 syntactically distinct goroutine leaks, for a total of 252 goroutine leak reports over 24 hours.
This proposal follows the implementation according to the available artifact, but with the following changes:
- The stacks of leaked goroutines (and other heap resources reachable only through the stacks) are not forcefully reclaimed by the GC; instead, once goroutines are flagged as leaked, they are treated as traceable by the GC, as in the runnable GC. This preserves the soundness of goroutine leak detection without changing what the GC perceives as reachable memory.
- Goroutine leak detection is triggered on-demand by extracting a new type of profile in the
runtime/pprofpackage:goroutineleak. More specifically, requesting a goroutine leak profile triggers a special goroutine leak detection GC cycle which marks leaked goroutines. These goroutines are then collected separately. - In the current implementation, changes are enabled via the
goroutineleakprofileflag forGOEXPERIMENT. (runtime/pprof,runtime: add new goroutine leak profile as a GOEXPERIMENT #75280)
Activity
[-]proposal: runtime: Deadlock detection by using the garbage collector[/-][+]proposal: runtime: deadlock detection by using the garbage collector[/+]seankhliao commentedon Jul 14, 2025
I think this would benefit from an implementation before a proposal.
Proposal golang#74609: goroutine leak detection by using the garbage …
VladSaiocUber commentedon Jul 15, 2025
Implementation provided at #74622. Waiting for CLA resolution.
Proposal golang#74609: goroutine leak detection by using the garbage …
randall77 commentedon Jul 15, 2025
Not sure where Gaby is.
Related:
#18835
#13759
VladSaiocUber commentedon Jul 16, 2025
@randall77 Thanks for the reference to #13759! Indeed, the proposed approach is very similar to what is proposed in this comment.
gopherbot commentedon Jul 16, 2025
Change https://go.dev/cl/688335 mentions this issue:
Proposal #74609: goroutine leak detection by using the garbage collectorProposal golang#74609: goroutine leak detection by using the garbage …
mknyszek commentedon Jul 25, 2025
To start with, I want to highlight a few aspects of this proposal that I think are great.
All of these things together, to me, make this proposal very exciting, and I personally think we should move forward with this in general.
Feedback
With that being said, let's quibble about some details.
For context, we have discussed this proposal to some degree in the performance and diagnostics meeting on July 17th: #57175 (comment).
Naming
Firstly, I think most of us agreed this should be called a "goroutine leak detector" or "goroutine leak finder" (I prefer the former slightly since it aligns with "race detector").
API
I think the main thing that came out of there is that we think the clearest way to expose this new functionality is as a new type of profile. That is a very clear expression of intent on the part of a Go developer that they would like to perform this analysis. It's like a subset of a goroutine profile.
(One point that came up in the July 17th discussion was possibly exposing this as part of a goroutine profile by using labels. My counter-point to this is that I think it buries the functionality a little too deep. We could make it a parameter on goroutine profiles, but that's less discoverable, and labels are less obvious in the tooling. A new kind of profile is unambiguous in the tooling and more discoverable.)
The good news is although this still requires a proposal, it doesn't need to change any APIs to start with. We just need to give it a name so that the
runtime/pprofpackage will understand it.Additional interfaces
That being said, I think for this new functionality to really shine, we should consider two more ways to interface with it (that could be their own proposals, or tacked on to this one).
net/http/pprofshould have an endpoint for this so that it can be used at scale.go testshould possibly have a flag to make it easy to add this sort of checking during regular testing.For (1): I think this is uncontroversial and unobjectionable. We want to help tooling perform this analysis on running services at scale, and I think the popularity of
net/http/pprofsuggests we should support this functionality there too as a new endpoint. We just need to pick some names, otherwise it mostly follows the example set byFor (2): Much like the race detector, being able to opt-in to it from
go testis very valuable, especially for uses of Go at smaller scales. The idea is that the detector would run just before the test binary exits. The biggest question about exposing the leak detector as ago testflag is what it should do. It could either behave more like the profiling flags and generate a file, or it can behave more like the race detector and force the program to exit with a message and a non-zero exit code. From our discussion on July 17th, I think it's clear to all of us that the leak detector sits in a space in between the race detector and a profile and both might be useful. Unfortunately I do not think we can just enable it by default, because it might break too many programs (it's also not clear to me that crashing is compliant with the Go memory model).GOEXPERIMENT
I think we should probably leave the
GOEXPERIMENTin there, but only as a feature flag to disable the runtime integration in case we find there's something problematic about it.Alternatives
Collecting dead goroutines and the memory they reference
IIRC, @VladSaioc's previous implementation would actually collect dead goroutines and memory only that goroutine would reference. @VladSaioc has reported that a slightly more invasive and slightly more complex implementation is fast enough to run on every GC cycle for many workloads, so reclaiming dead goroutines is feasible.
This is attractive but has some problems. The main concerns from Go team members came down to the fact that memory is often not the only resource leaked by a leaked goroutine (for example, network connections, files, C memory, etc.). These could be handled by finalizers/cleanups, but it's also unclear exactly whether finalizers/cleanups are really safe to run in these scenarios. As a result, we worry that reclaiming only the memory can just push back the problem, creating new, harder-to-debug problems down the line.
Given these objections, it seems safest to start with a detector rather than a reclamation mechanism. We can always return to these questions in the future if we have new information, or a new way to approach these problems. A goroutine leak detector is still very valuable to the Go ecosystem.
(As a final point here, the faster version of the analysis still requires additional GC STWs that might impact latency and make the GC less able to parallelize its work. This might be fixable, or we could work around it by only executing the analysis occasionally, for example every 100 GC cycles or something.)
Proposal golang#74609: goroutine leak detection by using the garbage …
41 remaining items
VladSaiocUber commentedon Apr 22, 2026
@aclements I have a plan for implementing it. Just to reiterate, the exception is only for the main goroutine if it is specifically blocked at
select{}.aclements commentedon Apr 22, 2026
@VladSaiocUber , great! And, yes, that's right.
VladSaiocUber commentedon Apr 23, 2026
@aclements Change implemented here #78922. Gerrit CL: https://go-review.googlesource.com/c/go/+/770020#message-421140e1814b6f17353d2df6cd20f40967fccdb8
runtime: exclude main goroutine blocked on select{} from goroutine le…
gopherbot commentedon May 6, 2026
Change https://go.dev/cl/774621 mentions this issue:
internal/goexperiment,runtime: drop goroutineleakprofile experimentgopherbot commentedon May 6, 2026
Change https://go.dev/cl/774620 mentions this issue:
internal/buildcfg: enable goroutineleakprofile GOEXPERIMENT by defaultinternal/buildcfg: enable goroutineleakprofile GOEXPERIMENT by default
internal/goexperiment,runtime: drop goroutineleakprofile experiment
gopherbot commentedon May 8, 2026
Change https://go.dev/cl/775621 mentions this issue:
internal/goexperiment: actually delete goroutineleakprofile experimentinternal/goexperiment: actually delete goroutineleakprofile experiment
gopherbot commentedon Jun 9, 2026
Change https://go.dev/cl/788882 mentions this issue:
_content/doc/go1.27: document goroutine leak profile_content/doc/go1.27: document goroutine leak profile