runtime: performance and diagnostics meeting notes

Since 2022, members of the Go team and the Go community have met every two weeks to discuss Go runtime topics. Until 2025 the focus was mainly on diagnostics, but we've also delved into plenty of other topics related to the runtime as well. To reflect this, I'm renaming this group to the "performance and diagnostics biweekly meeting."

We currently have a virtual sync every 2 weeks, Thursdays at 11 AM NYC time. Please ping me at mknyszek -- at -- golang.org for an invite if you're interested in attending. This issue will be updated regularly with meeting notes from those meetings.

The reasons for starting this meeting in 2022 are still relevant today. So for the sake of some nostalgia, and also as a reminder of the importance of diagnostics to making Go an effective tool for our users, I leave behind some of the original text from this GitHub issue below.

As the Go user base grows, more and more Go developers are seeking to understand the performance of their programs and reduce resource costs. However, they are locked into the relatively limited diagnostic tools we provide today. Some teams build their own tools, but right now that requires a large investment. This issue extends to the Go team as well, where we often put significant effort into ad-hoc performance tooling to analyze the performance of Go itself.

This issue is a tracking issue for improving the state of Go runtime diagnostics and its tooling, focusing primarily on runtime/trace traces and heap analysis tooling.

@mknyszek

2022-12-07 Sync

Attendees: @mknyszek @aclements @prattmic @felixge @nsrip-dd @rhysh

Notes:

pprof labels in execution traces
- Michael K: I need to follow up on the CL and issue.
- Michael P: Have you considered runtime/trace regions?
- Nick: Yes, but it doesn't quite hit our use-cases.
- Rhys: Could use logs instead. Is the point to follow goroutines?
- Felix: Yeah.
- Rhys: Started parsing the trace format for some internal tooling.
- Felix: Prefer labels because inheritance. Mostly like it for profile tools on top, but maybe doesn't matter for tracing tools. The parent goroutine gets recorded so things like request IDs can be tracked in post-processing.
  - Also useful for attributing CPU time.
- Michael P: Tasks are inherited, but inheritance is processed downstream.
  - Would be nice to have pprof labels just so you don't have to think about which one to use.
- Michael K: Useful even just to bridge pprof and runtime/trace.
- Austin: Agreed. We can one day get to the point of deprecating the old APIs as well.
- Rhys: RE: Attributing CPU time, can see at better than 10 ms of granularity already (even if Ps aren't CPU time, it's still time the rest of the app couldn't run).
- Michael P: There's an issue about measure CPU time on-line. (TODO: Find it.)
Trace parsing API
- Michael K: How important is this? Priority?
- Felix: Important for the community, but we can't make use of it in our case. Will the trace format change in between releases?
- Michael K: I think we can always guarantee a trace format for a release.
- Michael P: How high level should this API be?
  - cmd/trace has two levels:
    - Low-level that understands format and events.
    - Higher-level that understands relationships between goroutines, etc.
- Michael K: Page trace splits this into "parser" and "simulator." The latter is more stateful.
- Felix: Intuitive feeling toward lower level API.
- Rhys: +1 to low level.
- Austin: Scalability of processing traces.
  - Currently not in a good state in low or high level format (currently requires the whole trace).
  - Can fix trace wire format for low-level parsing scalability issues, but it's much less clear how to do this for the high-level format.
- Austin: Flight recorder idea.
  - Interacts interestingly with simulation. Current trace snapshots everything.
  - Solved this in debuglog; reads its own tail and keeps local state updated.
  - Complicated trade-offs in this space.
- Felix: We use a lot of JFR, one thing that's nice is it's broken down into self-contained chunks.
Michael K sent out a very half-baked trace format revamp. (Thanks for the comments! Far from ready to share more widely.)
- The next step is to measure the actual current overhead.
  - Maybe add a mode to Sweet?
  - Rhys: Have been collecting CPU profiles and execution traces. 20% of CPU time during execution trace is for execution trace itself. 95% of overhead is collecting stack traces.
    - Collect 1 second every 5000 seconds and no one complains. People do complain about goroutine profiles every 2 minutes.
  - Michael K: Shooting for KUTrace overhead, so making stack traces optional/faster is just step 1.
  - Felix: Trace effect on tail latency.
    - Rhys: Traces are my view of tail latency.
  - Felix: Benchmark for pathological cases and worst case.
  - Austin: Linked to trace overhead issue, Dmitry proposed switching to frame pointer unwinding.
  - Felix: At some point implemented frame pointer unwinding in userland and it was 50x faster (link).
  - Rhys: Not sure what kind of tool you could build without stack traces in an app that doesn't set pprof labels, tasks, regions, trace logs, etc.
  - Michael K: Integration with structured logging?
  - Michael P: It does add yet another way to add things to runtime/trace.
  - Rhys: The standard library (e.g. database/sql) doesn't currently use runtime/trace at all, maybe it should.
  - Michael K: This connects to deciding what goes into a trace. I think this is a very good idea.
  - Felix: +1. Java world and JFR does this.
Deciding what goes into a trace
- Disabling stack tracing / reduce stack trace depth
- Filtering by pprof labels
- Specific event classes
- Standard library events
- Rhys: I've made this decision for my organization. Expected that you do profiling for a long running service. No opportunity for app owners to express opinions. People who complained forked the package, turned it off, and now coming back. I kind of want everything.
- Felix: I would love to be in a place where we can do that, but we get pushback from users when the overhead is too high.
- Rhys: The question is how close we get within 1% overhead. My choice was to get everything, but less often.
- Felix: Desire to get more of everything is in conflict with adding more kinds of things in the trace.
- Michael P: Agreed. Ideally we have tracing that's sufficiently fast that we have on all the time, but if libraries are allowed to add new traces then it could be a problem. It would be nice to turn that off without forking a library.
Before next sync:
- Michael K: Unblock pprof labels patch and benchmarking trace overhead.
- Felix: I can contribute a worst case benchmark.
  - Currently blocked on pprof labels in trace.
- Felix: Started to work on gentraceback. Might work on it over the holidays.
  - Trying for bug-for-bug compatibility.
- Michael P: Austin has been working on this too.

I'll miss the Dec 22nd meetup because I'm traveling for the holidays. That being said, if I find time I might also look into #57159 . Getting a proof of concept for Perfetto UI integration (ideally using their protocol buffer format) is probably more important than the gentraceback refactoring at this point. I just tried to work with a 300 MB (15s of prod activity) yesterday, and it was a real eye opener to the way the current UI struggles.

I don't know if it's relevant (probably nothing new for the folks on this thread), but I had similar problems with the go tool trace viewer where it would freeze on me all the time, esp. in the per-goroutine view (/trace?goid=N). I figured out you can download perfetto-compatible JSON data from /jsontrace?goid=N. (/jsontrace gives the default view). This can then be uploaded to ui.perfetto.dev. This doesn't show all the information in the trace so it's not as great, but I was glad to have something that worked.

would the pprof labels also show up in goroutine traces?

I'm working on a PoC that improves native stack unwinding on Windows by adding additional information to the PE file. This will help debugging with WinDbg and profiling with Windows Performance Analyzer. Would this work fit into the effort tracked by this issue?

@thediveo

@thediveo I think that might be a good question for #56295, or you could file another issue. Off the top of my head, that doesn't sound like it would be too difficult to do.

@qmuntal Oh neat! That's awesome. I think it's a little tangential to the work we're proposing here, unless you also plan to do anything with the runtime's unwinder (i.e. gentraceback). Then again, if one of the goals is better integration with the Windows Performance Analyzer that's certainly more in the same spirit. Do you have an issue for tracking that already?

Do you have an issue for tracking that already?

I still have to prepare the proposal, I plan to submit it next week.

unless you also plan to do anything with the runtime's unwinder (i.e. gentraceback).

Not for now, but once I finish this I want to investigate how feasible is too unwind native code and merge it with the Go unwinding, in case the exception happens in a non-Go module.

Do you have an issue for tracking that already?

I do now #57302 😄

Change https://go.dev/cl/459095 mentions this issue: sweet: add support for execution traces and measuring trace overhead

@mknyszek

2022-12-22 Sync

Attendees: @mknyszek @aclements @prattmic @bboreham @rhysh @dominikh

Organizational stuff
- OK to record meetings?
- Meeting recorded with transcript this week (please ask if you would like to see it).
Trace overhead benchmarks
- https://go.dev/cl/459095

name                                old time/op            new time/op            delta
BiogoIgor                                      17.7s ± 3%             17.5s ± 4%     ~     (p=0.190 n=10+10)
BiogoKrishna                                   15.1s ± 4%             15.1s ± 4%     ~     (p=0.739 n=10+10)
BleveIndexBatch100                             5.78s ± 7%             5.76s ±11%     ~     (p=0.853 n=10+10)
BleveQuery                                     2.37s ± 0%             2.37s ± 0%   -0.26%  (p=0.016 n=8+10)
FoglemanFauxGLRenderRotateBoat                 16.9s ± 9%             16.9s ± 7%     ~     (p=0.796 n=10+10)
FoglemanPathTraceRenderGopherIter1             36.7s ± 1%             44.4s ± 2%  +21.01%  (p=0.000 n=10+10)
GoBuildKubelet                                 47.0s ± 2%             48.8s ± 3%   +3.72%  (p=0.000 n=10+10)
GoBuildKubeletLink                             8.89s ± 2%             8.88s ± 4%     ~     (p=0.720 n=10+9)
GoBuildIstioctl                                45.9s ± 1%             47.8s ± 2%   +4.09%  (p=0.000 n=10+10)
GoBuildIstioctlLink                            9.07s ± 2%             8.99s ± 2%     ~     (p=0.095 n=10+9)
GoBuildFrontend                                15.7s ± 4%             16.1s ± 2%   +2.45%  (p=0.043 n=10+10)
GoBuildFrontendLink                            1.38s ± 2%             1.37s ± 3%     ~     (p=0.529 n=10+10)
GopherLuaKNucleotide                           27.9s ± 0%             27.9s ± 1%     ~     (p=0.853 n=10+10)
MarkdownRenderXHTML                            256ms ± 2%             256ms ± 2%     ~     (p=1.000 n=9+9)
Tile38WithinCircle100kmRequest                 618µs ± 7%             657µs ±10%   +6.30%  (p=0.015 n=10+10)
Tile38IntersectsCircle100kmRequest             722µs ± 6%             773µs ± 4%   +6.96%  (p=0.000 n=10+9)
Tile38KNearestLimit100Request                  508µs ± 3%             532µs ± 3%   +4.73%  (p=0.000 n=10+10)

name                                old average-RSS-bytes  new average-RSS-bytes  delta
BiogoIgor                                     68.8MB ± 2%            71.8MB ± 4%   +4.40%  (p=0.000 n=10+10)
BiogoKrishna                                  4.42GB ± 0%            4.42GB ± 0%     ~     (p=0.739 n=10+10)
BleveIndexBatch100                             194MB ± 2%             198MB ± 3%   +1.91%  (p=0.008 n=9+10)
BleveQuery                                     536MB ± 0%             537MB ± 1%     ~     (p=0.190 n=10+10)
FoglemanFauxGLRenderRotateBoat                 444MB ± 1%             446MB ± 0%   +0.41%  (p=0.035 n=10+9)
FoglemanPathTraceRenderGopherIter1             132MB ± 1%             142MB ± 4%   +7.61%  (p=0.000 n=10+10)
GoBuildKubelet                                1.75GB ± 1%            1.85GB ± 1%   +5.51%  (p=0.000 n=10+10)
GoBuildIstioctl                               1.35GB ± 1%            1.42GB ± 1%   +5.49%  (p=0.000 n=10+9)
GoBuildFrontend                                511MB ± 2%             543MB ± 1%   +6.31%  (p=0.000 n=10+9)
GopherLuaKNucleotide                          37.0MB ± 1%            40.4MB ± 2%   +9.24%  (p=0.000 n=9+10)
MarkdownRenderXHTML                           21.8MB ± 3%            24.0MB ± 3%  +10.14%  (p=0.000 n=9+8)
Tile38WithinCircle100kmRequest                5.40GB ± 1%            5.38GB ± 1%     ~     (p=0.315 n=10+10)
Tile38IntersectsCircle100kmRequest            5.72GB ± 1%            5.71GB ± 1%     ~     (p=0.971 n=10+10)
Tile38KNearestLimit100Request                 7.26GB ± 0%            7.25GB ± 0%     ~     (p=0.739 n=10+10)

name                                old peak-RSS-bytes     new peak-RSS-bytes     delta
BiogoIgor                                     95.9MB ± 4%            98.5MB ± 3%   +2.70%  (p=0.030 n=10+10)
BiogoKrishna                                  4.49GB ± 0%            4.49GB ± 0%     ~     (p=0.356 n=9+10)
BleveIndexBatch100                             282MB ± 3%             284MB ± 4%     ~     (p=0.436 n=10+10)
BleveQuery                                     537MB ± 0%             538MB ± 1%     ~     (p=0.579 n=10+10)
FoglemanFauxGLRenderRotateBoat                 485MB ± 1%             483MB ± 0%     ~     (p=0.388 n=10+9)
FoglemanPathTraceRenderGopherIter1             180MB ± 2%             193MB ± 3%   +7.19%  (p=0.000 n=10+10)
GopherLuaKNucleotide                          39.8MB ± 3%            46.0MB ±20%  +15.56%  (p=0.000 n=9+10)
MarkdownRenderXHTML                           22.1MB ± 3%            25.5MB ± 7%  +15.45%  (p=0.000 n=9+10)
Tile38WithinCircle100kmRequest                5.70GB ± 1%            5.68GB ± 1%   -0.45%  (p=0.023 n=10+10)
Tile38IntersectsCircle100kmRequest            5.93GB ± 1%            5.91GB ± 2%     ~     (p=0.631 n=10+10)
Tile38KNearestLimit100Request                 7.47GB ± 1%            7.46GB ± 0%     ~     (p=0.579 n=10+10)

name                                old peak-VM-bytes      new peak-VM-bytes      delta
BiogoIgor                                      802MB ± 0%             803MB ± 0%   +0.11%  (p=0.000 n=10+10)
BiogoKrishna                                  5.24GB ± 0%            5.24GB ± 0%   +0.01%  (p=0.001 n=10+10)
BleveIndexBatch100                            1.79GB ± 0%            1.79GB ± 0%   +0.05%  (p=0.000 n=8+8)
BleveQuery                                    3.53GB ± 1%            3.53GB ± 1%     ~     (p=0.237 n=10+10)
FoglemanFauxGLRenderRotateBoat                1.21GB ± 0%            1.16GB ± 4%     ~     (p=0.163 n=8+10)
FoglemanPathTraceRenderGopherIter1             875MB ± 0%             884MB ± 0%   +1.02%  (p=0.000 n=10+10)
GopherLuaKNucleotide                           733MB ± 0%             734MB ± 0%   +0.11%  (p=0.000 n=9+10)
MarkdownRenderXHTML                            733MB ± 0%             734MB ± 0%   +0.10%  (p=0.000 n=10+9)
Tile38WithinCircle100kmRequest                6.42GB ± 0%            6.39GB ± 1%     ~     (p=0.086 n=8+10)
Tile38IntersectsCircle100kmRequest            6.62GB ± 1%            6.61GB ± 2%     ~     (p=0.927 n=10+10)
Tile38KNearestLimit100Request                 8.16GB ± 1%            8.18GB ± 0%     ~     (p=0.649 n=10+8)

name                                old p50-latency-ns     new p50-latency-ns     delta
Tile38WithinCircle100kmRequest                  144k ± 3%              159k ± 3%  +10.56%  (p=0.000 n=9+9)
Tile38IntersectsCircle100kmRequest              215k ± 1%              232k ± 2%   +7.91%  (p=0.000 n=9+10)
Tile38KNearestLimit100Request                   347k ± 2%              373k ± 1%   +7.21%  (p=0.000 n=10+10)

name                                old p90-latency-ns     new p90-latency-ns     delta
Tile38WithinCircle100kmRequest                  908k ± 6%              956k ± 9%   +5.22%  (p=0.043 n=10+10)
Tile38IntersectsCircle100kmRequest             1.07M ± 4%             1.11M ± 5%   +4.33%  (p=0.001 n=10+10)
Tile38KNearestLimit100Request                  1.03M ± 3%             1.05M ± 4%   +2.64%  (p=0.011 n=10+10)

name                                old p99-latency-ns     new p99-latency-ns     delta
Tile38WithinCircle100kmRequest                 7.55M ± 9%             7.93M ±13%     ~     (p=0.089 n=10+10)
Tile38IntersectsCircle100kmRequest             7.81M ± 8%             8.39M ± 2%   +7.36%  (p=0.000 n=10+8)
Tile38KNearestLimit100Request                  2.03M ± 4%             2.08M ± 5%   +2.52%  (p=0.019 n=10+10)

name                                old ops/s              new ops/s              delta
Tile38WithinCircle100kmRequest                 9.73k ± 7%             9.16k ±11%   -5.83%  (p=0.015 n=10+10)
Tile38IntersectsCircle100kmRequest             8.31k ± 6%             7.77k ± 4%   -6.55%  (p=0.000 n=10+9)
Tile38KNearestLimit100Request                  11.8k ± 3%             11.3k ± 3%   -4.51%  (p=0.000 n=10+10)

Introduction: Bryan Boreham, Grafana Labs
- Questions within the team about whether useful information has been derived from Go execution traces.
- Phlare: continuous profiling. Interested in linking together various signals (distributed tracing, profiling)
- Michael K: Interesting data point about usability.
- Michael P: Hard to link application behavior to trace.
- Bryan: Example: channels. Still don't really know where to find that data.
- Dominik: One of the reasons I started on gotraceui was to surface more information and do more automatic inference and analysis of the data.
- Rhys: Execution trace technique: get data out of them to find the interesting traces. Try to extract features that would be interesting up-front.
  - Starts with internal trace parser. Have code to find start and end of HTTP requests, DNS lookups, etc.
  - Tooling on the way to get open sourced.
Heap analysis plan (proposal: x/debug: make the core and gocore packages public #57447)
- Austin: Additional context is we're confident in the API we're planning to export, as opposed to tracing which we have nothing for yet.
https://go.dev/issue/57307 proposal: cmd/trace: visualize time taken by syscall
- Austin: Does Perfetto do better with instantaneous events?
  - Michael P: Yes, there's a 20px wide arrow but we have so many.
  - Rhys: Hold shift, draw a box. If you aim well, you get what you want.
- Rhys: Why is there only one timestamp on some events?
  - Austin: We can add another timestamp.
  - Michael P: Syscall fast path does a lot less.
pprof labels in traces
- Michael K: I think I've unblocked Nick. Michael and I are reviewing.
runtime.gentraceback cleanup
- Austin: Back and forth on the issue about making it an iterator, sent out CLs, not tested yet.
Next meeting: Jan 5th, Michael P and Michael K won't be here, so Austin will run it.
Action items:
- We're slowing down for the holidays, so no strong expectations
- Michael K:
  - Try to land execution trace benchmarking.
  - Might look into heap analysis stuff.
  - After break, might want to start working on trace format more seriously.
Happy holidays!

2026-03-12 sync

Attendees: Michael P, Michael K, Jake, Nick, Bryan, Chris, Dominik, Vlad, Carlos

Chris: Any plans for supporting collecting delta profiles?
- https://go.dev/issue/74545 proposal: new Recorder types in runtime/pprof
- Nick: Had a working block and mutex implementation of the proposal. Heap profile is more difficult. For the live heap profile, we show where something was allocated. If you want an accurate representation of the live heap, you need to include allocations from before you created the recorder.
- Michael P: The live heap profile is really a “snapshot” profile, not delta profile, because it shows the live heap at one point in time. You do indeed either need to record from the beginning for that, or put unknown on a bunch of the heap (assuming we snapshot at the end of a delta profile).
- Michael K: You could occasionally snapshot the live heap profile and drop anything that has cancelled out.
- Michael K: We could manage the live heap profile in epochs. Every second, we produce a new read only snapshot that is the base for future collection.
- Chris: Actually do want continuous profiling from the start. The issue is more the size of the profile. We really just want the deltas of the profiles, we can reconstruct that ourselves.
- Michael P: Wondering if the snapshotting would still work well for an exploding worst case. I think it might as long as we actually drop the memory.
- Bryan: Grafana has a client library that remembers the last profile you downloaded and does a delta. But it does cost more. https://github.com/grafana/pyroscope-go/tree/main/godeltaprof
- Michael P: I think net/http/pprof does this too.
- Nick: net/http/pprof does it very inefficiently. At DataDog we've tried to push this. Grafana does even better with linknames. For the Recorder proposal, we might also want to make use of a snapshot API so that net/http/pprof could still use it out of the box.
- Milind: RE: epochs, what if the API is general enough that you can specify with respect to which epoch you want to collect the deltas. But I can imagine that someone wants to ask for a few epochs behind.
- Michael P: I think with the Recorder API it is possible to do it yourself.
- Milind: I was thinking about something more general. But it might require more memory. You could maybe come up with a really clever data structure. But yeah, there's probably no need for it.
- Michael P: One thing this API doesn't provide is deltas of live heap profiles.
Nick: Questions about goroutine leak profile. It flags permanently blocked goroutines.
- Vlad: We could add an exemption for that.
- Nick: In theory people might do it in main to get it to block so it doesn't exit.
- Vlad: I was also wondering on a syntactic level why that was added to the language.
- Michael K: It's a profile so you could filter it out. Or you can work around it by blocking on a global channel.
- Michael P: Plenty of APIs don't have a "wait to exit," and if you call those APIs in main your only option is to select.
- Vlad: It does also slightly degrade the precision of the leak detector to do this.
- Michael P: We could keep the analysis but just filter out these goroutines.
- Vlad: My rationale for supporting it was also thinking that you might want to know what your dependencies are doing.
- Michael K: I also worry a bit about beginners' experience.
- Vlad: We can also make it clearer that certain goroutines have chosen to block forever.
- Michael K: Could use goroutine labels?
- Nick: For DataDog, we know that this case just uses runtime.block.
- Michael P: Or notice that goroutine ID 1 is always the main goroutine.

2026-03-26 sync

Attendees: Jake, Nick, Dominik, Michael P, Michael K, Carlos, thepudds, Chris

Nick: Recorder API
- Posted additional comments: proposal: new Recorder types in runtime/pprof #74545 (comment)
- Should Stop take the Writer instead of Start?
- It’s awkward to have two different heap profile APIs.
- Michael K: I was thinking that they would have the same implementation, but change the default sample index. Some people don’t realize that sample index exists at all. The existing pprof API does this for “heap” vs “allocs”.
- Nick: If that is the only difference, perhaps it should just be a configuration option on a “Memory” recorder.
- Michael K: I don’t like the name “Memory” because it only includes heap memory, unless we decide to start including things like goroutine stacks.
- Nick: They need a snapshot method to be usable for replacing the net/http/pprof implementation.
- Michael K: We could make changes only for net/http/pprof/v2. Get rid of the idea that you can get a cumulative profile from start.
- Michael K: Most profiles don’t make much sense from program start. Heap makes the most sense because we have it on by default, plus the live heap portion needs it to be on as an implementation detail.
- Nick: I would like to have only a single internal implementation even if we don’t expose the API.
- Chris: Curious will this new API work for Mutex profile as well? I think it's also accumulative for now?
- Nick: Yes, prototype works. Heap profile prototype almost working.
- Michael P: I think we are going to need to do something like the generation/snapshot idea from last time where alloc/frees that cancel out are dropped. I don’t think it is acceptable for a new API to introduce “unknowns” into profiles with no way out. Perhaps if you set MemProfileRate to 0 then you can get “unknowns”.
- Michael K: If someone sets MemProfileRate to 0 midway through execution, I don’t think we need to support that use case.
- Michael P: Are we saying that the background snapshotting is mandatory and there is no way to get live heap at all with that disabled (even with “unknowns”).
- Michael K: I think so? Implementation wise, when disabled we could still attach profiling specials, but only use them to update a “sampled live heap size” counter, which solves the growing memory usage problem.
- Michael P: Regardless, once we can drop old samples, the memory usage problem is mostly mitigated.
- Michael K: Nick, would you like to take over the proposal?
- Nick: Sure.
- <Frustration at GitHub “Load more”>
Jake: map performance
- Michael P: My plan is to enable GOEXPERIMENT=mapsplitgroup in Google production to see if there is a negative, neutral, or positive performance impact
- thepudds: The GOEXPERIMENT is handy to allow folks to test if there are regressions in 1.27.
- Michael P: I was planning to make a decision before 1.27 and remove the GOEXPERIMENT, but perhaps we should keep the GOEXPERIMENT regardless of which way we select.
- Michael K: We may want to ask for feedback from the CockroachDB folks.
- thepudds: my recollection is the original swisstable authors chose KVKVKVKV... and that was after many experiments. If that layout was selected as “better” for the C++ version, some Go people might have regressions with changing to KKKKKVVVVV.
Michael K: tsgo OOMs
- Jake: It was likely our issue.
- Jake: I’m interested in some of the work on dynamic GC limits to help with this.
- Michael K: I don’t think that is going to fix this problem. The GC is always going to need some form of proportional runway if the live heap is large.
- Jake: If someone sees their language server using 40GB of memory, it is actually using 20GB of live memory. It is usually pretty static because you load the entire project into memory at start and then only make small changes (1 files) later.
- Michael K: You are still allocating. It’s important to ensure you don’t get into death spirals. GOGC helps keep GC costs consistent regardless of heap size. GOMEMLIMIT is the control to accept higher cost to maintain a lower limit. You could come up with a GOMEMLIMIT adjustment policy.
- thepudds: but if GOGC=50 moves the memory from 40GiB to 30GiB, and people are not complaining about CPU today, that could be something that most people would view as decent improvement (in that example, could be -10GiB, w/ only minor CPU increase in absolute terms).
- Jake: Folks don’t mind CPU usage since tsgo is so much faster. Users just don’t like seeing a big memory number in top.
- Jake: Setting GOMEMLIMIT=1GB seems to be using 1GB all the time, despite keeping GOGC=100.
- Michael K: I’d love to see a GC trace. Setting GOMEMLIMIT without changing GOGC should have no behavior change until you get close to the limit.
- Michael P: This feels like a textbook case where you want to lower GOGC to trade more CPU time for less memory usage. You could also choose to lower only after the initial loading stage.
- Michael K: This could be a good use case for defensive setting GOMEMLIMIT. Default to ~80% of the system available memory, and then periodically check RSS and lower GOMEMLIMIT if possible. The API is intended to be called frequently. This is a good case where we could learn best practices for CLI tools.
- Jake: Parallel processing vs serial in the node version does mean more is going on at once, so peak memory usage potential is higher.
- Michael K: When you have a big spike and know things will be lower afterwards is the one good use for debug.FreeOSMemory
- thepudds: Is once every 30s OK?
- Michael K: It is an O(heap size) operation that is very expensive, so it’s hard to say. The GC might be running every minute. Once every 50 GCs might be a better heuristic, though that is difficult to implement. If you can avoid using it you will save headaches.
- thepudds: There’s probably “good, better, best” solutions here. GOGC=50 will give you an easy 25% drop in memory with a little more CPU usage (but CPU usage sounds like it is in great spot currently, so maybe a good and easy trade off here).
- thepudds: Maybe just always setting GOMEMLIMIT one time at startup to ~75% of system memory (or 50%, or pick some fixed percent) would also be a stopgap measure. I think that means the CPU can go up if it then later passes that (up to 25% of total system cores? 50% of total system cores? I forget, but there’s a cpu limit it tries to respect if it can’t keep it below GOMEMLIMIT).
- Michael K: in that scenario, it would allow up to 50% of system cores
- thepudds: That could be a simpler near-term step than being more sophisticated about changing GOMEMLIMIT over time dynamically with some feedback loop.
- thepudds: Doing that that might help most people (without any complaints), but for the people who drive their memory very high, it could change the complaints from “it ate all my memory and took out my whole laptop at a very bad time”, to instead be “it caused my fans to turn on and use more CPU than I’d like”, which is I think less frustrating for a user than taking out the whole laptop due to using all mem.

2026-04-09 sync

Attendees: Dominik, Chris, Carlos, Felix, Vlad, Nick, Bryan, Michael K, Michael M, thepudds

Nick: Recorder API update
- Slow going. 90% done, now I need to do the other 90%, repeatedly. Zeno's paradox.
- Conceptually dense, but not that much code.
Vlad: Goroutine leak profile blog post
- Going through more review rounds, looking good.
Felix: ~RSS Memory Profile
- New idea: break down the newly allocated memory using the alloc profile data.
- Michael K: One issue is sampling of large allocations. Another issue is that allocated memory (not in the in-use profile) is dead.
- Felix: Aggregating data across a fleet of services helps with the sampling issue. Reducing allocated memory (when there is no memory limit) will reduce the heap goal.
- Bryan: Another issue is RSS < Go Memory, when allocating memory without touching it.
- Many: Yeah… virtual memory.
- Michael K: We could do something with smaps.
- Michael K: I like all this. Ship it! What would be the blast radius? Tests?
- Felix: I'll start a GitHub issue and maybe look into a prototype.
- Nick: DataDog has a live heap metric that is derived by using the live heap profile. There might be a communication problem to address in rolling this out.
- Bryan: Let's also not lose sight of the original intent: an RSS profile. We'd want to make sure that other stuff ends up in the profile, like estimates of non-Go memory from platform RSS metrics.
- Felix: I have a document but let me lay out the specifics. Here's what I'm thinking: you have a pseudo-frame at the bottom for "Go memory" which is "total - heap released." Then any difference from OS RSS would be some separate "system memory." And then under Go memory would be the specifics, the breakdown of memory from runtime/metrics. Profiling data slides into categories where it makes sense.
- Felix: There might be some skew.
- Nick: Calling runtime.GC right before is probably good enough.
- thepudds: There'd be a lot of value in having the OS-level CPU usage and OS-level RSS as runtime metrics (which is mostly orthogonal to mem profile discussion). It's been proposed and rejected because it's annoying to do across all platforms. But it was also before runtime/metrics existed. And it can be best effort outside of the main 3 platforms (linux/mac/windows). I think justifiable even just from triage perspective on issue tracker. I dusted off an older experiment based on some of the typescript-go discussion a month or so ago (where part of that
- discussion was around “wait, what mem stat are you using?” for different users). Could start as a GOEXPRIMENT.
- FreeBSD folks might show up and fill in support later, or might not.
- Michael K: SGTM. Send a proposal! Send a CL! The runtime/metrics proposals get through pretty quickly.
- Felix: +1 to the runtime/metrics proposals going through quickly.
Chris: People have been getting confused about the mutex profile. The mutex profile monotonically accumulates data. What's the best way for people to visualize and interpret it? People don't really expect it to just accumulate. What you really want is a rate or a derivative.
- Felix: Internally we do delta profiles. But people still tend to be misused or struggle to understand. Usually when something goes wrong the mutex profile also spikes. And people tend to misattribute issues to the mutex profile. The execution tracer helps a lot to paint a clearer picture. Nick's Recorder API should be better on that.
- Nick: +1 to Felix's. What also confuses people is why the mutex profile shows unlock. We had to document a lot and explain that it represents the critical section. It's been hard to communicate.
- Michael K: Mutexes are just hard. Such simple things, so much emergent behavior.
- thepudds: Ian Lance Taylor wrote some nice prose about why RWMutex is a usually a bad idea. Perhaps we should document that.
  - Recent related discussion around essentially warning people about RWMutex via the doc in issue sync: RWMutex Readers can starve writers for many seconds #76808.
  - I bounced that conversation just now to see if anyone is interested in taking a stab at a doc update:
    sync: RWMutex Readers can starve writers for many seconds #76808 (comment)

2026-04-23 sync

Attendees: Michael P, Carlos, Vlad, Dominik, Nick, Chris, thepudds, Felix, Michael K, Jake Bailey

Vlad: https://go-review.googlesource.com/c/go/+/770020 added exemption for main goroutine blocked on select{}
- Vlad: We treat the goroutine as leaked for analysis purposes, but just exclude it from the profile.
Felix: Updates on RSS Profile type (early CLs).
- Felix: Show everything the Go runtime counts in runtime/metrics.
- Felix: The non-in-use heap memory shows the allocations from the memory that was just swept (freed). I think we can count this without additional memory.
- Michael: So this profile is effectively a snapshot at the end of a GC cycle.
- thepudds: In the non-live heap memory a single number, or broken down?
- Felix: it is broken down.
- thepudds: for the metric values, wouldn’t you have to keep a three element rolling buffer (or maybe 2?) of the metrics in a staggered cycle ring buffer to allow for the metric values to be consistent with the mem profile data when it is later written (given that the mem profile data is already up to 2 GC cycles stale when written)?
- Felix: good point, I would need that
- Felix: Non-heap memory is also included.
- thepudds: would it work then to use the metric data to fill in the flame graph?
- Felix: Yes, but I want to fill in the non-live-heap space in the flame graph with something more useful since it will be ~50% of the view.
- thepudds: that makes sense for more detail, though maybe worth recording the simpler approach as a less desired option
- Michael: What is the primary motivation of including non-live-heap stack traces? There isn’t fundamentally new information here, allocs profile can include similar information.
- Felix: Both UX (half the view being empty is annoying), plus including the useful allocs information.
- thepudds: Could that be filled in with post-processing using the allocs profile? (Maybe more complicated or less precise, but probably worth at least listing as a rejected alternative)
- Michael: cgo memory is just RSS - the Go stuff
- Felix: Yes, but it gets a bit dicier because RSS may not quite match Go’s world.
- Michael: Crazy idea is runtime.SetCgoMemProfile to call into the malloc profiler to fill in the cgo data.
- thepudds: I think you are planning on sharing a doc?
- Felix: Yes
Michael P: proposal: runtime: support a percentage for GOMEMLIMIT #78912
- Michael P: Idea is GOMEMLIMIT=80% of system memory limit, or cgroup memory limit.
- Felix: Lots of users do this in practice, but manually. Convenience would be nice.
- Felix: For mixed C binaries, it is hard to know what percentage to set. Setting a percentage of total RSS may be useful.
- Michael K: I am worried about trying to read memory limits cross platform.
- Michael K: Why is automemlimit not sufficient?
- Michael K: e.g., how do OSes report total memory if there is swap?
- Michael P: We count total online CPUs for GOMAXPROCS cross platform.
- Michael P: For cgroups, do we use hard limit or soft limit.
- Michael K: OS memory subsystems tend to change more than CPU, so long-term compatibility is a concern. Particularly for Windows and Darwin, which don’t even matter that much. More complicated than the group cpu limit, and likely to be harder for Go project to keep up with OS changes and variation for memory.
- Felix: It would be nice, but I’m not sure it should be high priority given all these hard questions.
- Michael K: This is purely opt-in, lowering the priority.
- Michael K: In the past, someone who did not set GOMAXPROCS could have gotten into more trouble with the defaults (part of justification for cgroup cpu limit work already landed), vs. someone who leaves the default GOGC at 100 is in reasonably good spot.
Jake: tsgo on macOS with Mac FUSE was exceeding the 10k thread limit.
- Michael P: This is not a bug in the runtime because the runtime can’t know if a syscall is required to unblock another system call. The canonical fix is a goroutine pool or semaphore to limit maximum concurrency. Alternatively, increase the 10k thread limit (Google does this).
- Jake: Don’t want to increase thread limit on users’ machines
- Felix: We’ve had things that look like memory leaks with tons of cgo calls because glibc’s allocator is not very aggressive with releasing memory.
- Michael K: If the os.File can go through netpoll then it wouldn’t need to block a thread.
- Michael P: I’m not sure what it looks like on Darwin, but we might not use netpoll for files because we don’t expect them to block.
- Michael P: In fact, even on Linux, I don’t think FUSE files can go through netpoll because they are normal files, so they are always “ready to read”. It’s more like having a very slow spinning disk.

@prattmic

2026-05-07 sync

Attendees: @prattmic, @jakebailey, @bboreham, @nsrip-dd, Arnav, Kshitij, @matloob, @cagedmantis, @VladSaiocUber, @felixge

Michael P: Freeze for go1.27 is in 2 weeks.
Arnav: runtime: scavenger unable to release idle heap on ARM64 with 64KB OS pages #78962 runtime: scavenger unable to release idle heap on ARM64 with 64KB OS pages
- Arnav: Timeseries DB (m3db) on arm64 saw significant increase in memory on arm64 vs amd64.
- Arnav: Not a consistent increase, some instances +10% vs amd64, some +40%.
- Arnav: This application has large heap, lots of allocations, lots of different size classes.
- Arnav: Spans are smaller than page size, so even if a span is completely empty, the page may not be.
- Kshtjij: This is the only application hurt by 64KB pages, so we don’t want to reduce page size.
- Kshitij: We ran into this problem with jemalloc. We were able to tune the size of per-thread caches.
- Michael P: I’m sure we can make some improvements, though I’m not sure how much. Go hasn’t been tuned for 64KB pages at all.
- thepudds: would be great to add details on what was seen/improved with jemalloc to the issue (might be concrete source of ideas/data)
- Milind: Can we tune the allocator at the command line?
  Michael P: No, all the “tunables” are constants in the runtime.
- Michael P: Changing the size of the span might be relevant, but is probably not straightforward.
Felix: Proposal for RSS profile type is now on GH: proposal: runtime/pprof: add a (rss) memory profile type #79179
- Issue 1: Go Memory > RSS: Scale down?
  - Michael P: scaling the numbers feels a bit odd, it could be confusing. It’s an annoying UX problem, ideally we’d record the real data and decide when displaying it.
  - Felix: We could have separate sample type to display both ways. If RSS forces scaling, we’d include a warning.
  - thepudds: How much are you tied to pprof for UI vs some alternative tool interpreting.
  - Felix: The pprof tool is the primary concern.
  - thepudds: Will the second sample type always be included, or only when necessary (which I guess would mean it might appear/dissapper over time for a given process)
  - Felix: I was thinking only when necessary, but it could always be included, just not useful.
- Issue 2: Naming: “rss” or “memory”?
  - Felix: I’d like this to eventually be the default memory profile type.
  - Michael P: We don’t really have a “default” memory profile, just “heap”. So we could use “memory” to designate a preferred default.
  - Felix: Windows doesn’t call it “rss”, making the name a bit odd.
  - Nick: This isn’t strictly a superset of the heap profile, since it doesn’t include allocs.
  - Felix: I don’t know if want to keep including allocs in all the memory profiles.
  - Felix: On the other hand, separate profiles duplicate the symbol table.
  - Bryan: When asking users for debugging information, it’s nicer to be able to ask for only one profile.
  - Felix: From that perspective, this could just be a new sample type in the heap profile. “rss” profile type would just change the default sample type.
- Issue 3: Compressed Memory: Use uncompressed RSS metric from OS?
  - Felix: All OSes provide an uncompressed RSS value, and we should use that.
  - Michael P: +1
- Issue 4: Snapshot consistency: Small price on STW okay for runtime metrics? Small inconsistency okay with RSS?
  - Felix: We need a copy of a bunch of runtime/metrics. Is it OK to add to STW?
  - Michael P: I’m not worried about it, but we should measure it.
  - Felix: RSS syscall during STW?
  - Michael P: A syscall on Linux should be fine from a correctness point of view, I’m not sure about platforms like darwin that call into libc. But I concerned about a syscall that might be expensive during STW.
  - Michael P: Perhaps do it before mark termination STW?
  - thepudds: This could delay the critical path. After mark termination is probably more conservative.
  - thepudds: Allocation can also go to the OS and increase heap size (e.g., large allocation that is outside the size classes), so I think probably racing user allocations as well, at least in some corner cases. That said, timing window is tiny, probably fine, and the other noise is probably bigger.
  - Felix: regardless this is better than anything users could do today.
- Issue 5: Platform support: Is it okay to only support first class ports? Unsupported platforms could simply not have a virtual “RSS” frame.
  - Porting policy: clarify Go support policy for secondary ports #53383
  - Michael P: It is fine to leave things to secondary port maintainers.
Vlad: Racelite comparison with TSan on https://github.com/uber-research/drfix/blob/main/data/skeletons
- Vlad: We are working on a lightweight TSAN alternative using watchpoints. We have some data comparing this standard TSAN. There is still instrumentation, but lightweight.
- 10s: racelite find ~18%, TSAN 85.3%
- 5min: racelite finds ~75%, TSAN 85.7%
- racelite finds a few races that TSAN doesn’t.
- Vlad: Unsure if this is a bug in TSAN, or just unlucky.
- From real-world races from https://dl.acm.org/doi/10.1145/3729265
- thepudds: my initial racelite version padded the heap allocations, but later version uses less memory (something like 0.3% or 0.5% mem overhead).
- Vlad: Memory use is vastly improved vs standard TSAN, which was our main blocker.
- Michael P: I still have dreams of making this work without any instrumentation, as that has the best UX.
- Milind: Same, but lack of PEBS in cloud feels like a big blocker.
Bryan: Experiment with Transparent Huge Pages: https://github.com/bboreham/go-huge
thepudds: improving starting stack size determination
- have uber folks been able to get data using new/temp debug flags? Or is that maybe unlikely?
- also, I have a more aggressive alternative we could discuss briefly
Jake: Where does GOEXPERIMENT=mapsplitgroup stand?
- Michael P: Due to delays in Google release, we didn’t complete the experiment. Next cycle!

@prattmic

2026-05-21 sync

Attendees: @prattmic @cagedmantis @nsrip-dd @dominikh @mauri870

Nick: pprof Recorder API
- Nick: Object special records are in a linked list, can we keep a pointer directly to the recorder data structure? That can’t be on the heap, but that means we need some kind of reference counting to know when it is safe to free.
- Sampled heap object [on heap] -> special record [off heap] -> profile recorder][on heap]
  - One special per enabled recorder (when sampled)
Mauri: Need 128-bit atomics
- LLVM TSAN needs update for 128-bit atomics, but s390 specifically doesn’t support 128-bit atomics in TSAN
- https://go.dev/issue/61236 Accepted proposal for Uint64Pair
- Michael P: It should be fine for s390 to use a fallback or just be broken until TSAN is fixed. It is a secondary port.

@prattmic

2026-06-04 sync

Attendees: @prattmic @nsrip-dd @mauri870 @felixge @cagedmantis @matloob @bboreham @thepudds Chris

Nick: Sarah, a new intern at Datadog is working on bringing frame pointers to the primary unwinder.
- Needs review for 1.27: https://go-review.googlesource.com/c/go/+/730200
thepudds: What's the policy for bug fixes during the freeze.
- Michael P: Bug fixes for the regressions are generally accepted.
- The bar becomes raised the closer to the release we are.
- Fixes for bugs in new features are generally accepted up to the release.
- The closer we are to the release, the more we ask if something would be / should be included in a point release.
Mauri: Have a CL for CAS128. Most of the stack is ready. Still waiting on TSAN changes.
- TSAN changes are not yet merged in LLVM.
- Michael P: OpenBSD TSAN is out of date and IIRC removed from LLVM, so it cannot be updated.
- Michael P: Perhaps OpenBSD has to use 64-bit fallbacks.
Mauri: Proposal for atomic max and min https://go.dev/issue/63999

@thepudds

2026-06-18 sync

Attendees: @thepudds @mauri870 @cagedmantis @bboreham @jakebailey @felixge Chris, Milind

carlos: The first Go 1.27 RC1 should be released today or early next week.
- thepudds: it would be good to give an update on the release candidate date on the -dev mailing list on rough estimate, especially if it is expected to be next week
- carlos: We have been heads down working on release blockers internally.
mauri: FEAT_MOPS for hardware-accelerated memory copy/set on arm64
- cmd/asm: add support for arm64 FEAT_MOPS instructions #79994
- It should be faster. Mauri is seeking access to these machines (Neoverse V3 cores, maybe Google Axios/Azure Cobalt 200/Graviton 5)
- Prep work for 128-bit atomics is approved and set to wait-release.
  - Align128 compiler marker go.dev/cl/790320
  - CASPAL assembler support go.dev/cl/790321
    - Folks at Vitess were using CASP and found issues with synchronization because of the acquire/release semantics, they really needed CASPAL, which is coming up for 128-bit atomics on 1.28. cmd/asm: add ordered CASP variants (CASPA, CASPL, CASPAL) for arm64 #79929
    - No 16-byte alignment will likely bite them
    - There is an alternate implementation cmd/asm: add ordered CASP variants for arm64 #79930
mauri: CL for using epoll_pwait2 netpoll changes for finer granularity
- runtime: use epoll_pwait2 where available for sub-millisecond timeout precision on Linux #53824
  - Improving the latency for sub-milisecond timers increases kernel wakeups and thus CPU usage. Clearly we need a tradeoff.
- Felix: Consider a way to fallback to the old behavior or alternate behavior. GODEBUG tied to go version could help manage the risk of improving it
- thepudds: fyi, sometimes the perf builders can be very noisy. Commented more on the runtime: use epoll_pwait2 where available for sub-millisecond timeout precision on Linux #53824 issue on how to deal with that.
mauri: have a CL that improves string / byte compare on arm64 go.dev/cl/786560
chris: Is there any update or a timeline of the profiling work that Felix is working on?
- They are trying to decide on some proposals they would like to make.
- felix: Nick is working on it. He believes he is making good progress. He will follow up with his progress soon.
Milind: Provide more of a background to the question asked by Chris.
- They are looking at the case when in distributed systems. They are ...
felix: Mentions looking at flight recording.
milind: What is the overhead?
felix: describes how many resources might be needed -- often 10s of MiBs (not GiBs, not KiBs)
thepudds: There are some questions that have been raised in the proposal for the RSS memory profile proposal.
felix: Thanks thepudds for mentioning the questions. They will be sure to follow up.
thepudds: Asks Jake for a follow up on the memory usage.
jake: Has been very busy with other issues and has not had an opportunity to follow up.
- He is going to explore dropping the GOGC to 50.
- They've made other changes to release memory that isn't being used. It's using less memory depending on how you measure it. They have not had the chance to explore it. They've acknowledged that measuring these changes is difficult with customers.

@prattmic

2026-07-16

Attendees: @prattmic @mauri870 @bboreham @dominikh @nsrip-dd @cagedmantis Pawel, Chris, Krishna, Angelina

FYI Tree open for 1.28!
Mauri:
- Changes ready to submit:
  - cmd/internal/obj/arm64: add CASPA/CASPAL/CASPL pair atomic instructions go.dev/cl/786560
  - cmd/compile: add align128 compiler marker for 128-bit struct alignment go.dev/cl/790320
  - internal/bytealg: use NEON for compare_arm64.s large-input path go.dev/cl/786560
- Proposal approved: sync/atomic: add Max/Min operators sync/atomic: add Max/Min operators #63999
- LLVM Upstream, need 128bit atomics, pending review [tsan][go]: add a128 store, load, compare_exchange llvm/llvm-project#196833
- Epoll_pwait2 for netpoller runtime: use epoll_pwait2 where available for sub-millisecond timeout precision on Linux #53824
  - Michael P: Linux has timer “slack” (batching), I believe 250us by default.
  - Michael P: Austin has thought about batching.
  - thepudds: If there is only one timer, the higher precision doesn’t cause additional wakeups. So batching only makes sense if there are multiple timers in the same bucket.
  - Mauri: Most people don’t care about sub-millisecond timers.
  - thepudds: Chris Hines has cared a lot about timers in the past, may be worth reaching out to.
  - Michael P: Most users probably set a longer timer (10ms) and then it expires. If there are a bunch, they expire close together. Maybe actual sub-ms timers are kept precise, but 10ms timers are batched.
  - thepudds: Maybe the first timer is precise, and later timers are batched on top.
  - Mauri: I think we should be able to make this work for both cases.
  - Mauri: I’d rather avoid GODEBUG to avoid maintaining two paths.
  - Dominik: Would it really be that bad to make the user choose the desired precision/batching in the time.Timer case, i.e., add API and default to the sensible thing?
  - Mauri: What about time.Sleep, etc?
  - Dominik: I reckon if you care about the precision, you can use the API that supports it.
  - thepudds: Re: GODEBUG, perhaps the new path has a parameter that effectively behaves like the old code path.
  - Mauri: We don’t really know how often problems will occur.
  - Michael P: It’s not obvious how much this would be a problem. It’s not a compatibility issue, so not obvious. The better our heuristics, the less likely we need a GODEBUG.
  - thepudds: Could be internal GODEBUG.
Pawel / Krishna: https://go.dev/cl/793780 runtime: cache grown goroutine stacks to reduce stack-growth cost
- Pawel: 4% of Uber CPU in stack growth. Discussed 4 months ago. One idea was to increase starting stack size to a higher percentile.
- Pawel: Didn’t work. Too many small background goroutines. Also, the GC runs every few seconds, but many goroutines die too quickly to affect the average.
- Pawel: Alternative: Cache more goroutine stacks, anything smaller than 128kb. This worked very well, reduced cost 95%. Increases memory usage by ~8MB per P. Another risk is too many stacks could be cached between GCs, blowing up memory usage.
- Pawel: thepudds has an alternative low/high watermark CL, but we haven’t tested in prod.
- Pawel: We are using the one line patch for easier maintenance.
- Michael P: This seems fairly reasonable, I will reach out to Keith.
- Pawel: Per-P cache is bounded to 64, we could remove the global cache.
- Michael P: I wonder how much the global cache actually ends up being used.
- thepudds: Even simpler is to change the starting stack size constant.
- Pawel: We tried that, but total memory cost was much higher because background goroutines got big stacks.
- thepudds: The current implementation measures the in use stack size during the GC, not the peak.
- thepudds: My CL keeps a high watermark on how much stack growth happens on “young” goroutines. Measures during GC and when the goroutine dies. GODEBUG can print periodic statistics. It would be interesting to get some testing on it.
Michael P: racelite without instrumentation or emulation idea courtesy of @aclements.
- Add metadata to mark every load/store instruction in the binary.
- Set a periodic timer.
- When the timer fires, set breakpoint on the next load/store instruction (or instructions, if there is control flow*). Then continue execution to get there.
- If breakpoint never fires, no problem, just set another after the next timer.
- Michael P: I wonder if we should have a proposal sooner rather than later.
- Michael P: “GWP-TSAN” is a C++ implementation of this general idea.
- Dominik: Of course the bottleneck on LLVM won't go away if we support both race detectors.
- thepudds: I think proposal would be useful. We might want to give control of sampling rate/”aggressiveness”, for example, maybe via a 1-100 number or some other mechanism, or maybe that’s not needed, but could start to be discussed.
Chris: PGO basic block reordering status.
- Michael P: Shelved for now.
- Pawel: Does Google use PGO?
- Michael P: Services can opt in, exact improvement varies.
- Pawel: We did see some regressions.
- Nick: Datadog uses pgo. We had seen some regressions although I don't recall the exact issue. Some services had a few percent improvement, but like Michael said it's hard to clearly identify improvement from noise
- Bryan: I tried PGO in the past on Prometheus; did not see any improvement.
- thepudds: Try testing with the linker -randlayout flag for microbenchmarks if you see a degradation. Easy to do and often gives some quick insight.

@prattmic

2026-07-30

Attendees: @prattmic @cagedmantis @JunyangShao @mauri870 @nsrip-dd @jakebailey @bboreham Chris, Krishna, Milind

Mauri:
- openbsd/riscv64 is broken, can’t use runtime.stackcheck because stack hi/lo is not set. go.dev/cl/804961 reverts the change
- 128-bit atomics
  - needs a full race syso rebuild
  - Cas128 CL go.dev/cl/776400
    - Export Uint64Pair struct from internal/runtime/atomic or keep Cas128 operating on built-in types?
- sync/atomic Max/Min sync/atomic: add Max/Min operators #63999
  - C++26 will have max/min atomics too, great timing!
  - LLVM sanitizer support for the Go race detector, also C++26 [tsan] add fetch_max, fetch_min, fetch_umax, fetch_umin llvm/llvm-project#211388
  - cmd/internal/obj/arm64: add LDSMAX, LDSMIN, LDUMAX, LDUMIN instructions
Michael: Running mapsplitgroup experiment in Google.
Nick: Intern working on using FP in main unwinder
- Almost done, found most of the corner cases.
- Michael: Exciting, we’ll likely keep finding corner cases for a while.

fix(slides): adversarial verification fixes — 5 factual corrections

@prattmic

2026-08-13

Attendees: @prattmic @cagedmantis @rhysh @dominikh @jakebailey @nsrip-dd @felixge @thepudds Chris, Milind

Felix: proposal: runtime/pprof: add a (rss) memory profile type
- Updated proposal
- Make it the new default_sample_type for “heap”
- Felix: If it a new profile type (“memory”), it will be hard to find. Instead make it the default sample type in the “heap” profile.
- Michael: I like the idea of including in the heap profile. If we want more feedback, we could add it as non-default in one release and make default later. Not sure if necessary, though.
- Nick: The pprof web UI doesn’t make it obvious there are multiple sample types, maybe we can improve the UI.
- Jake: it's hard enough for me to get an agent to look at alloc_count, which is usually the thing that points towards bad code
- Felix: We add virtual root frames to stack traces, so none of the stacks will be shared with the other sample types, so none of the stacks will be shared. The pprof will definitely be bigger, though exact amount varies.
- thepudds: Changing the default here might have some precedence w/ the go 1.11 behavior change (not exactly the same, but an example of a change that happened and was worthwhile): https://go.dev/doc/go1.11#runtimepprofpkgruntimepprof
- Rhys: We are ending up with both the fast and slow path of the size-specialized malloc code in the heap profile, so we’ve already got a doubling (soon).
- Michael: I thought we skipped runtime frames?
- Nick: We drop the frames when serialize, but don’t merge the stack entries.
- Michael: Maybe we can include the virtual frames in inuse/alloc?
- Felix: The top “RSS” frame doesn’t make much sense.
- Felix: I will polish up my prototype.
amedee: Sharded Counters proposal proposal: sync: add new type Counter, a sharded counter #80645
- Carlos: Proposal is up. This is a byproduct of trying to make a generic sync.Sharded work. Counters are the most common use-case. It’s easier to make a performant dedicated Counter type rather than making something like sync.Sharded[uint64] competitive.
  - sync.Sharded proposal proposal: sync: support for sharded values #18802 updated with these challenges.
- Carlos: Please give the x/exp prototype a try.
Carlos: Also interested in feedback on whether folks are interested in more performant sync.RWMutex on large machines.
- Rhys: Discussed RWMutex with Austin, the tradeoff of optimize for readers vs making it as cheap as Mutex for writers. Perhaps most uses of RWMutex are wrong in ways that haven’t bitten the authors yet?
- Carlos: RWMutexes are supposed to be read heavy.
- Rhys: Are dramatic performance changes to RWMutex effectively a “compatibility” issue?
- Carlos: It’s difficult to keep the same semantics.
- Michael: I agree that we should optimize towards read-heavy. How to we help users move towards a better synchronization primitive for their use case?
- thepudds: it probably reasonable guess that the majority of RWMutex uses are wrong. Some related discussion in: sync: RWMutex Readers can starve writers for many seconds #76808
  - Documentation CL (needs feedback / review, so feel free to comment there!): https://go.dev/cl/767041
Nick: FP stack unwinder
- Internship wrapping up, will start sending CLs.
- CPU profiles can see stack shrinking when the stack is partially adjusted and the frame pointers are wrong.
- runtime.Callers gets ~80% faster.
- Felix: Compiler now uses the frame pointer on return via LEAVE instruction.
- Michael: Some assembly points LR to a loop header instead of a link address, crashing CPU profiling. We likely want to bail out.
- Nick: Maybe LR can be a valid scratch sometimes, can we do better?
Milind: racelite update
- Tested prototype on a real internal service.
- racelite matched the race detection rate, with way less CPU (~-17%) / memory overhead (~8x).
- racelite memory is much lower, and flat, TSAN memory grows over time.
- Detection rate similar (racelite slightly higher).
- TSAN has goroutine ancestors, racelite does not.
- Detected the same races.
- Want to deploy to subset of prod workloads.
- Jake: “Runtime Verification” (company name) does predictive race detection of races that could have happened but didn’t. I may have some thoughts on ways to make this work for Go.
- Milind: Motivation is to have a way to detect races on real production traffic. TSAN overhead is so high that this isn’t very workable.
- thepudds: Was the CPU result vs no instrumentation or vs TSAN?
- Milind: Local testing was ~30x vs TSAN, ~2x vs no instrumentation.
- Jake: Is this toggleable at runtime?
- Milind: Instrumented at compile time, so every load and store checks if it needs to do more work, but the probability of doing work is tunable.
- Felix: +1 to no instrumentation, we would use quite widely. A separate build would hurt adoption. One team even wants to disable TSAN in CI due to cost.
- thepudds: There is a spectrum of overhead vs precision. Some of the instrumentation design vs user API design vs runtime behavior (sleep strategy, parameters, etc.) could happen in parallel. Maybe something higher overhead is worth landing as a GOEXPERIMENT?
- thepudds: My newer prototype is 5-10% overhead of instrumentation (when race detection disabled), which is might be more amenable than 2x.
- Felix: 5% is probably too much more perf sensitive users for our biggest services, but they might still run short-term experiments to flush out some races they’ve been tracking down. Smaller services would be fine with 5%. We have more smaller services than our biggest services.

@prattmic

2026-09-10

Attendees: @prattmic @matloob @thepudds @thatnealpatel @felixge @nsrip-dd @randall77 @dominikh @cagedmantis Chris, Thomas, Krishna

Thomas: cmd/compile: removal of duffcopy reduced visibility of memory copying cost in profiles
- Looking at time spent copying memory. Used time spent in duffcopy as measure.
- Was able to save lots of CPU using this approach.
- Go 1.26 removes duffcopy, so now copy time is hidden in functions.
- It’s much harder to identify these costs.
- Prototype: tags copies as part of a fake inlined function so they appear in profiles.
- Alternative: Match PCs in profiles with assembly.
- Michael P: I’d love to see this kind of information, perhaps in DWARF?
- Keith: What other cases would we want? Divide is a good case, but is a dedicated instruction. Copy cases typically use large 128bit+ move. So perhaps a profile broken down by instruction type?
- Felix: Would skew be a problem?
- Michael P: I expect so for single instructions / small copies.
- Thomas: Looked for instruction patterns matching the duffcopy loop. Seems to work well, but more involved than normal because the data isn’t simply in pprof.
- Felix: pprof format now supports column number, it might help folks more generally without being tightly coupled to this problem.
- Michael P: We don’t include the column number, we’ve considered adding it for PGO.
- Felix: We often don’t have the binary at all.
- Felix: We could prototype adding column numbers.
- Chris: Raw binary might be very large? Can we include only the parts of the binary that were accessed?
Michael M: Cgo without a c toolchain proposal: cmd/cgo: cgo without a C toolchain #81450
- Future proposal (later today)
- Current cgo: run C compiler, use DWARF to see type layout, and then generate Go binding.
- What if we pre-ran the C compiler to generate the bindings so that users don’t need the C toolchain?
- Michael P: How do we handle things like macros that vary slightly between toolchains?
- Michael M: The hope is that things remain consistent within a single platform.
- Felix: This could be a general problem, but if we write the C library, we can make sure not to do weird things.
- Michael P: Maybe a problem for libc, but maybe not. I agree that we can say sufficiently weird C libraries are not supported.
- Michael M: I’d like feedback on the alternative proposal, using a alternative non-Go syntax.
- thepudds: Are people having problems with cgo today and does this sound useful?
- Felix: We have several use cases for SDKs where we want a shared core across languages. They’ve been using purego and will be excited to remove it.
Nick: Frame pointers in the primary unwinder for arm64 and amd64. CL stack starting at https://go.dev/cl/823985. Subsequent CLs enable frame pointers for callers/gcallers (~70% benchmark improvement), GC stack scanning, goroutine and CPU profiling, runtime lock profiling. CI is green! RE: CPU/goroutine profiling (CL), I have a memory model-related question: for stack copying, I want the new stack's internal pointers to be fixed before the goroutine's stack pointer is updated, so that the CPU/goroutine profiles don't see stale frame pointers. Is there any guarantee that another thread will see an up-to-date stack given that the stack pointer is updated after adjusting the stack frames? Or do we need explicit synchronization?
- Keith: You need a barrier when starting the goroutine?
- Nick: Problem for CPU profiling and goroutine profiling.
- Michael P: Maybe use the scan bit to lock stack growing and goroutine profiling?
- Nick: Maybe also be a problem for CPU profiling if the stack shrinks during systemstack?
- Michael P: I don’t see a problem with the proposed semantics, though I’m not sure if that is the best way to solve this.
Krishna: https://go.dev/cl/819781 PGO parser to accept Linux perf profile as an input
- Michael P: Makes sense to me.
- thepudds: Please file issues with specifics on bad cases.
- Felix: Used across the Datadog fleet, doesn’t cause problems. Adoption amongst our customers is slow, probably because it’s difficult to measure fleetwide impact in a noisy environment.
- Chris: Performance measurement is really difficult especially without A/B experimentation. Maintenance is the primary issue, who knows how things work if something goes wrong? Worried about hypothetical crashes.
- Krishna: How are folks measuring impact?
- Felix: Initially we looked at the benchmarks for the hottest functions in the hottest services, which showed promising results. In production we looked at week-to-week usage, and normalized CPU usage by work completed. Each service took 30min+ of manual analysis.

@prattmic

2026-09-24

Attendees: @prattmic @nsrip-dd @rhysh @felixge @bboreham @cagedmantis @randall77 Chris, Pawel, Krishna, Thomas

Rhys: Duffcopy thoughts (cmd/compile: removal of duffcopy reduced visibility of memory copying cost in profiles #81392)
- DWARF isn’t a great fit, but it is difficult to get back to the binary just from the pprof file.
- I’ve included program name and git commit in profiles.
- For the Go ecosystem, maybe that is everything from go version -m?
- Go builds are reproducible, so it should be possible to recreate the binary from this?
- Felix: We don’t have the binary or source code, so that wouldn’t work for us.
- Chris: Would this work with bazel?
- Felix: I believe go version -m is not included in bazel by default?
- Rhys: Profiles are missing important metadata. It sounds like the counterproposals are for the profiles to include a huge amount of new metadata, approaching “the entire dwarf” or “the entire executable” (though I’m not sure anyone is recommending those in the extreme). I think a kB or two, enough to associate with data that’s stored elsewhere (the source code, a stash of compiled binaries, a stash of fat summaries of binaries) is what’s missing.
- Chris: With Grafana, we’re having Grafana remap via bazel to get the data they need. https://github.com/grafana/pyroscope/blob/074d4925c85a2451c752980ddcd17f2a93599f62/docs/sources/view-and-analyze-profile-data/line-by-line.md#example-go-standard-library-mapping it works for stdlib, internal source code, assembly with proper mapping. Essentially the profile is ingested to the backend and backend remap the binary with source code mapping plus the source code access itself via GitHub etc.
- Felix: We have an ebpf profiler which does upload DWARF from the machine, so we could use that. But it is really nice to have pprof as a self-contained file for ease of use.
- Felix: The ebpf profiler has a new build ID format (standardized in otel) that hashes the first and last 4096 bytes of the file.
- Michael P: You can register a linker build ID with some service as coming from some source. We should include linker build ID in pprof at minimum, but more complete version information would be even better.
- Chris: For Grafana, we actually index the binary ourselves, don’t have Grafana build it.
- Felix: The pprof file itself always includes the symbols, which comes from the Go binary pclntab.
Nick: RSS profile
- The proposal has gone back and forth on whether it should be in the heap profile or a separate profile type.
- Currently on new profile, to avoid duplicating frames.
- New profile is similar to live heap profile, but doesn’t include historical allocations. Does anyone use that?
- Michael P: I do use the alloc profiles sometimes, but I’m not sure if it should be included.
- Felix: I think it is useful for understanding GC behavior. I think you might end up needing to fetch both.
- Nick: I’ve seen 30%-100% size increase from adding the alloc profile.
- Michael P: These stacks don’t include virtual frames? That makes this pretty similar to the same problem of including RSS in the heap profile.
- Nick: No virtual frames, they would be awkward.
- Felix: We’re assuming we can’t change the pprof format or viewer.
- Nick: What would we gain from loosening these?
- Felix: A changed format could have a more complex stack frame representation to avoid duplication. A changed viewer could build virtual frames out of pprof labels. Though two samples with different labels must duplicate.
- Michael P: I’ve wondered for a while what we should do about the viewer. Contribute improvements upstream? Change go tool pprof to deviate more from upstream?
- Felix: I think it will be difficult to get what we want with just viewer changes, but we should measure overhead.
Krishna: adaptive stack size optimizations runtime: better adaptive stack sizing #77893
- thepudds has a patch to measure watermarks, we have another patch to post soon and discuss.
- Today Go only caches Gs with the smallest stack size, we will cache Gs with stacks up to 128kb.

Navigation Menu

Repository navigation

runtime: performance and diagnostics meeting notes #57175

2022-12-07 Sync

2022-12-22 Sync

114 remaining items

2026-03-12 sync

2026-03-26 sync

2026-04-09 sync

2026-04-23 sync

2026-05-07 sync

2026-05-21 sync

2026-06-04 sync

2026-06-18 sync

2026-07-16

2026-07-30

2026-08-13

2026-09-10

2026-09-24

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

runtime: performance and diagnostics meeting notes #57175

Description

Activity

mknyszek commented on Dec 8, 2022

2022-12-07 Sync

felixge commented on Dec 9, 2022

tbg commented on Dec 9, 2022

thediveo commented on Dec 9, 2022

qmuntal commented on Dec 9, 2022

mknyszek commented on Dec 9, 2022

qmuntal commented on Dec 9, 2022

qmuntal commented on Dec 14, 2022

gopherbot commented on Dec 21, 2022

mknyszek commented on Dec 22, 2022

2022-12-22 Sync

114 remaining items

prattmic commented on Mar 26, 2026

2026-03-12 sync

prattmic commented on Mar 26, 2026

2026-03-26 sync

prattmic commented on Apr 23, 2026

2026-04-09 sync

prattmic commented on Apr 23, 2026

2026-04-23 sync

prattmic commented on May 7, 2026

2026-05-07 sync

prattmic commented on May 21, 2026

2026-05-21 sync

prattmic commented on Aug 6, 2026

2026-06-04 sync

prattmic commented on Aug 6, 2026

2026-06-18 sync

prattmic commented on Aug 6, 2026

2026-07-16

prattmic commented on Aug 6, 2026

2026-07-30

prattmic commented on Sep 21, 2026

2026-08-13

prattmic commented on Sep 21, 2026

2026-09-10

prattmic commented on Sep 24, 2026

2026-09-24

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions