testing/synctest: new package for testing concurrent code

@gabyhelp

Current proposal status: #67434 (comment)

This is a proposal for a new package to aid in testing concurrent code.

// Package synctest provides support for testing concurrent code.
package synctest

// Run executes f in a new goroutine.
//
// The new goroutine and any goroutines transitively started by it form a group.
// Run waits for all goroutines in the group to exit before returning.
//
// Goroutines in the group use a synthetic time implementation.
// The initial time is midnight UTC 2000-01-01.
// Time advances when every goroutine is idle.
// If every goroutine is idle and there are no timers scheduled,
// Run panics.
func Run(f func())

// Wait blocks until every goroutine within the current group is idle.
//
// A goroutine is idle if it is blocked on a channel operation,
// mutex operation,
// time.Sleep,
// a select with no cases,
// or is the goroutine calling Wait.
//
// A goroutine blocked on an I/O operation, such as a read from a network connection,
// is not idle. Tests which operate on a net.Conn or similar type should use an
// in-memory implementation rather than a real network connection.
//
// The caller of Wait must be in a goroutine created by Run,
// or a goroutine transitively started by Run.
// If it is not, Wait panics.
func Wait()

This package has two main features:

It permits using a fake clock to test code which uses timers. The test can control the passage of time as observed by the code under test.
It permits a test to wait until an asynchronous operation has completed.

As an example, let us say we are testing an expiring concurrent cache:

type Cache[K comparable, V any] struct{}

// NewCache creates a new cache with the given expiry.
// f is called to create new items as necessary.
func NewCache[K comparable, V any](expiry time.Duration, f func(K) V) *Cache {}

// Get returns the cache entry for K, creating it if necessary.
func (c *Cache[K,V]) Get(key K) V {}

A naive test for this cache might look something like this:

func TestCacheEntryExpires(t *testing.T) {
	count := 0
	c := NewCache(2 * time.Second, func(key string) int {
		count++
		return fmt.Sprintf("%v:%v", key, count)
	})

	// Get an entry from the cache.
	if got, want := c.Get("k"), "k:1"; got != want {
		t.Errorf("c.Get(k) = %q, want %q", got, want)
	}

	// Verify that we get the same entry when accessing it before the expiry.
	time.Sleep(1 * time.Second)
	if got, want := c.Get("k"), "k:1"; got != want {
		t.Errorf("c.Get(k) = %q, want %q", got, want)
	}

	// Wait for the entry to expire and verify that we now get a new one.
	time.Sleep(3 * time.Second)
	if got, want := c.Get("k"), "k:2"; got != want {
		t.Errorf("c.Get(k) = %q, want %q", got, want)
	}
}

This test has a couple problems. It's slow, taking four seconds to execute. And it's flaky, because it assumes the cache entry will not have expired one second before its deadline and will have expired one second after. While computers are fast, it is not uncommon for an overloaded CI system to pause execution of a program for longer than a second.

We can make the test less flaky by making it slower, or we can make the test faster at the expense of making it flakier, but we can't make it fast and reliable using this approach.

We can design our Cache type to be more testable. We can inject a fake clock to give us control over time in tests. When advancing the fake clock, we will need some mechanism to ensure that any timers that fire have executed before progressing the test. These changes come at the expense of additional code complexity: We can no longer use time.Timer, but must use a testable wrapper. Background goroutines need additional synchronization points.

The synctest package simplifies all of this. Using synctest, we can write:

func TestCacheEntryExpires(t *testing.T) {
        synctest.Run(func() {
                count := 0
                        c := NewCache(2 * time.Second, func(key string) int {
                        count++
                        return fmt.Sprintf("%v:%v", key, count)
                })

                // Get an entry from the cache.
                if got, want := c.Get("k"), "k:1"; got != want {
                        t.Errorf("c.Get(k) = %q, want %q", got, want)
                }

                // Verify that we get the same entry when accessing it before the expiry.
                time.Sleep(1 * time.Second)
                synctest.Wait()
                if got, want := c.Get("k"), "k:1"; got != want {
                        t.Errorf("c.Get(k) = %q, want %q", got, want)
                }

                // Wait for the entry to expire and verify that we now get a new one.
                time.Sleep(3 * time.Second)
                synctest.Wait()
                if got, want := c.Get("k"), "k:2"; got != want {
                        t.Errorf("c.Get(k) = %q, want %q", got, want)
                }
        })
}

This is identical to the naive test above, wrapped in synctest.Run and with the addition of two calls to synctest.Wait. However:

This test is not slow. The time.Sleep calls use a fake clock, and execute immediately.
This test is not flaky. The synctest.Wait ensures that all background goroutines have idled or exited before the test proceeds.
This test requires no additional instrumentation of the code under test. It can use standard time package timers, and it does not need to provide any mechanism for tests to synchronize with it.

A limitation of the synctest.Wait function is that it does not recognize goroutines blocked on network or other I/O operations as idle. While the scheduler can identify a goroutine blocked on I/O, it cannot distinguish between a goroutine that is genuinely blocked and one which is about to receive data from a kernel network buffer. For example, if a test creates a loopback TCP connection, starts a goroutine reading from one side of the connection, and then writes to the other, the read goroutine may remain in I/O wait for a brief time before the kernel indicates that the connection has become readable. If synctest.Wait considered a goroutine in I/O wait to be idle, this would cause nondeterminism in cases such as this,

Tests which use synctest with network connections or other external data sources should use a fake implementation with deterministic behavior. For net.Conn, net.Pipe can create a suitable in-memory connection.

This proposal is based in part on experience with tests in the golang.org/x/net/http2 package. Tests of an HTTP client or server often involve multiple interacting goroutines and timers. For example, a client request may involve goroutines writing to the server, reading from the server, and reading from the request body; as well as timers covering various stages of the request process. The combination of fake clocks and an operation which waits for all goroutines in the test to stabilize has proven effective.

@gabyhelp's overview of this issue: #67434 (comment)

I really like how simple this API is.

Time advances when every goroutine is idle.

How does time work when goroutines aren't idle? Does it stand still, or does it advance at the usual rate? If it stands still, it seems like that could break software that assumes time will advance during computation (that maybe that's rare in practice). If it advances at the usual rate, it seems like that reintroduces a source of flakiness. E.g., in your example, the 1 second sleep will advance time by 1 second, but then on a slow system the checking thread may still not execute for a long time.

What are the bounds of the fake time implementation? Presumably if you're making direct system calls that interact with times or durations, we're not going to do anything about that. Are we going to make any attempt at faking time in the file system?

If every goroutine is idle and there are no timers scheduled, Run panics.

What if a goroutine is blocked on a channel that goes outside the group? This came to mind in the context of whether this could be used to coordinate a multi-process client/server test, though I think it would also come up if there's any sort of interaction with a background worker goroutine or pool.

or is the goroutine calling Wait.

What happens if multiple goroutines in a group call Wait? I think the options are to panic or to consider all of them idle, in which case they would all wake up when every other goroutine in the group is idle.

What happens if you have nested groups, say group A contains group B, and a goroutine in B is blocked in Wait, and then a goroutine in A calls Wait? I think your options are to panic (though that feels wrong), wake up both if all of the goroutines in group A are idle, or wake up just B if all of the goroutines in B are idle (but this block waking up A until nothing is calling Wait in group B).

How does time work when goroutines aren't idle?

Time stands still, except when all goroutines in a group are idle. (Same as the playground behaves, I believe.) This would break software that assumes time will advance. You'd need to use something else to test that case.

What are the bounds of the fake time implementation?

The time package: Now, Since, Sleep, Timer, Ticker, etc.

Faking time in the filesystem seems complicated and highly specialized, so I don't think we should try. Code which cares about file timestamps will need to use a test fs.FS or some such.

What if a goroutine is blocked on a channel that goes outside the group?

As proposed, this would count as an idle goroutine. If you fail to isolate the system under test this will probably cause problems, so don't do that.

What happens if multiple goroutines in a group call Wait?

As proposed, none of them ever wake up and your test times out, or possibly panics if we can detect that all goroutines are blocked in that case. Having them all wake at the same time would also be reasonable.

What happens if you have nested groups

Oh, I didn't think of that. Nested groups are too complicated, Run should panic if called from within a group.

This is a very interesting proposal!

I feel worried that the synctest.Run characteristic of establishing a "goroutine group" and blocking until it completes might make it an attractive nuisance for folks who see it as simpler than arranging for the orderly completion of many goroutines using other synchronization primitives. That is: people may be tempted to use it in non-test code.

Assuming that's a valid concern (if it isn't then I'll retract this entire comment!), I could imagine mitigating it in two different ways:

Offer "goroutine groups" as a standalone synchronization primitive that synctest.Run is implemented in terms of, offering the "wait for completion of this and any other related goroutines" mechanism as a feature separate from synthetic time. Those who want to use it in non-test code can therefore use the lower-level function directly, instead of using synctest.Run.
Change the synctest.Run design in some way that makes it harder to misuse. One possible idea: make synctest.Run take a testing.TB as an additional argument, and then in every case where the proposal currently calls for a panic use t.FailNow() instead. It's inconvenient (though of course not impossible) to obtain a testing.TB implementation outside of a test case or benchmark, which could be sufficient inconvenience for someone to reconsider what they were attempting.

(I apologize in advance if I misunderstood any part of the proposal or if I am missing something existing that's already similarly convenient to synctest.Run.)

The fact that synctest goroutine groups always use a fake clock will hopefully act as discouragement to using them in non-test code. Defining goroutines blocked on I/O as not being idle also discourages use outside of tests; any goroutine reading from a network connection defeats synctest.Wait entirely.

I think using idle-wait synchronization outside of tests is always going to be a mistake. It's fragile and fiddly, and you're better served by explicit synchronization. (This prompts the question: Isn't this fragile and fiddly inside tests as well? It is, but using a fake clock removes much of the sources of fragility, and tests often have requirements that make the fiddliness a more worthwhile tradeoff. In the expiring cache example, for example, non-test code will never need to guarantee that a cache entry expires precisely at the nanosecond defined.)

So while perhaps we could offer a standalone synchroniziation primitive outside of synctest, I think we would need a very good understanding of when it would be appropriate to use it.

As for passing a testing.TB to synctest.Run, I don't think this would do much to prevent misuse, since the caller could just pass a &testing.T{}, or just nil. I don't think it would be wise to use synctest outside of tests, but if someone disagrees, then I don't think it's worth trying to stop them.

Interesting proposal. I like that it allows for waiting for a group of goroutines, as opposed to all goroutines in my proposal (#65336), though I do have some concerns:

Complexity of implementation: Having to modify every time-related function may increase complexity for non-test code. Would it make more sense to outsource the mock time implementation to a third party library? The Wait() function should be sufficient for the third party library to function deterministically, and goroutines started by Run() would behave like normal goroutines in all aspects.
Timeouts: In my proposal, WaitIdle() returns a <-chan struct{} since it allows for a test harness to abort the test if it takes too long (e.g. 30 seconds in case the test gets stuck in an infinite loop). Would it make sense for the Wait() function here to return a chan too to allow for timeouts?

One of the goals of this proposal is to minimize the amount of unnatural code required to make a system testable. Mock time implementations require replacing calls to idiomatic time package functions with a testable interface. Putting fake time in the standard library would let us just write the idiomatic code without compromising testability.

For timeouts, the -timeout test flag allows aborting too-slow tests. Putting an explicit timeout in test code is usually a bad idea, because how long a test is expected to run is a property of the local system. (I've seen a lot of tests inside Google which set an explicit timeout of 5 or 10 seconds, and then experience flakiness when run with -tsan and on CI systems that execute at a low batch priority.)

Also, it would be pointless for Wait to return a <-chan struct{}, because Wait must be called from within a synctest group and therefore the caller doesn't have access to a real clock.

I wanted to evaluate practical usage of the proposed API.

I wrote a version of Run and Wait based on parsing the output of runtime.Stack. Wait calls runtime.Gosched in a loop until all goroutines in the current group are idle.

I also wrote a fake time implementation.

Combined, these form a reasonable facsimile of the proposed synctest package, with some limitations: The code under test needs to be instrumented to call the fake time functions, and to call a marking function after creating new goroutines. Also, you need to call a synctest.Sleep function in tests to advance the fake clock.

I then added this instrumentation to net/http.

The synctest package does not work with real network connections, so I added an in-memory net.Conn implementation to the net/http tests.

I also added an additional helper to net/http's tests, which simplifies some of the experimentation below:

var errStillRunning = errors.New("async op still running")

// asyncResult is the result of an asynchronous operation.
type asyncResult[T any] struct {}

// runAsync runs f in a new goroutine,
// and returns an asyncResult which is populated with the result of f when it finishes.
// runAsync calls synctest.Wait after running f.
func runAsync[T any](f func() (T, error)) *asyncResult[T]

// done reports whether the asynchronous operation has finished.
func (r *asyncResult[T]) done() bool

// result returns the result of the asynchronous operation.
// It returns errStillRunning if the operation is still running.
func (r *asyncResult[T]) result() (T, error)

One of the longest-running tests in the net/http package is TestServerShutdownStateNew (https://go.googlesource.com/go/+/refs/tags/go1.22.3/src/net/http/serve_test.go#5611). This test creates a server, opens a connection to it, and calls Server.Shutdown. It asserts that the server, which is expected to wait 5 seconds for the idle connection to close, shuts down in no less than 2.5 seconds and no more than 7.5 seconds. This test generally takes about 5-6 seconds to run in both HTTP/1 and HTTP/2 modes.

The portion of this test which performs the shutdown is:

shutdownRes := make(chan error, 1)
go func() {
	shutdownRes <- ts.Config.Shutdown(context.Background())
}()
readRes := make(chan error, 1)
go func() {
	_, err := c.Read([]byte{0})
	readRes <- err
}()

// TODO(#59037): This timeout is hard-coded in closeIdleConnections.
// It is undocumented, and some users may find it surprising.
// Either document it, or switch to a less surprising behavior.
const expectTimeout = 5 * time.Second

t0 := time.Now()
select {
case got := <-shutdownRes:
	d := time.Since(t0)
	if got != nil {
		t.Fatalf("shutdown error after %v: %v", d, err)
	}
	if d < expectTimeout/2 {
		t.Errorf("shutdown too soon after %v", d)
	}
case <-time.After(expectTimeout * 3 / 2):
	t.Fatalf("timeout waiting for shutdown")
}

// Wait for c.Read to unblock; should be already done at this point,
// or within a few milliseconds.
if err := <-readRes; err == nil {
	t.Error("expected error from Read")
}

I wrapped the test in a synctest.Run call and changed it to use the in-memory connection. I then rewrote this section of the test:

shutdownRes := runAsync(func() (struct{}, error) {
	return struct{}{}, ts.Config.Shutdown(context.Background())
})
readRes := runAsync(func() (int, error) {
	return c.Read([]byte{0})
})

// TODO(#59037): This timeout is hard-coded in closeIdleConnections.
// It is undocumented, and some users may find it surprising.
// Either document it, or switch to a less surprising behavior.
const expectTimeout = 5 * time.Second

synctest.Sleep(expectTimeout - 1)
if shutdownRes.done() {
	t.Fatal("shutdown too soon")
}

synctest.Sleep(2 * time.Second)
if _, err := shutdownRes.result(); err != nil {
	t.Fatalf("Shutdown() = %v, want complete", err)
}
if n, err := readRes.result(); err == nil || err == errStillRunning {
	t.Fatalf("Read() = %v, %v; want error", n, err)
}

The test exercises the same behavior it did before, but it now runs instantaneously. (0.01 seconds on my laptop.)

I made an interesting discovery after converting the test: The server does not actually shut down in 5 seconds. In the initial version of this test, I checked for shutdown exactly 5 seconds after calling Shutdown. The test failed, reporting that the Shutdown call had not completed.

Examining the Shutdown function revealed that the server polls for closed connections during shutdown, with a maximum poll interval of 500ms, and therefore shutdown can be delayed slightly past the point where connections have shut down.

I changed the test to check for shutdown after 6 seconds. But once again, the test failed.

Further investigation revealed this code (https://go.googlesource.com/go/+/refs/tags/go1.22.3/src/net/http/server.go#3041):

st, unixSec := c.getState()
// Issue 22682: treat StateNew connections as if
// they're idle if we haven't read the first request's
// header in over 5 seconds.
if st == StateNew && unixSec < time.Now().Unix()-5 {
	st = StateIdle
}

The comment states that new connections are considered idle for 5 seconds, but thanks to the low granularity of Unix timestamps the test can consider one idle for as little as 4 or as much as 6 seconds. Combined with the 500ms poll interval (and ignoring any added scheduler delay), Shutdown may take up to 6.5 seconds to complete, not 5.

Using a fake clock rather than a real one not only speeds up this test dramatically, but it also allows us to more precisely test the behavior of the system under test.

Another slow test is TestTransportExpect100Continue (https://go.googlesource.com/go/+/refs/tags/go1.22.3/src/net/http/transport_test.go#1188). This test sends an HTTP request containing an "Expect: 100-continue" header, which indicates that the client is waiting for the server to indicate that it wants the request body before it sends it. In one variation, the server does not send a response; after a 2 second timeout, the client gives up waiting and sends the request.

This test takes 2 seconds to execute, thanks to this timeout. In addition, the test does not validate the timing of the client sending the request body; in particular, tests pass even if the client waits

The portion of the test which sends the request is:

resp, err := c.Do(req)

I changed this to:

rt := runAsync(func() (*Response, error) {
	return c.Do(req)
})
if v.timeout {
	synctest.Sleep(expectContinueTimeout-1)
	if rt.done() {
		t.Fatalf("RoundTrip finished too soon")
	}
	synctest.Sleep(1)
}
resp, err := rt.result()
if err != nil {
	t.Fatal(err)
}

This test now executes instantaneously. It also verifies that the client does or does not wait for the ExpectContinueTimeout as expected.

I made one discovery while converting this test. The synctest.Run function blocks until all goroutines in the group have exited. (In the proposed synctest package, Run will panic if all goroutines become blocked (deadlock), but I have not implemented that feature in the test version of the package.) The test was hanging in Run, due to leaking a goroutine. I tracked this down to a missing net.Conn.Close call, which was leaving an HTTP client reading indefinitely from an idle and abandoned server connection.

In this case, Run's behavior caused me some confusion, but ultimately led to the discovery of a real (if fairly minor) bug in the test. (I'd probably have experienced less confusion, but I initially assumed this was a bug in the implementation of Run.)

At one point during this exercise, I accidentally called testing.T.Run from within a synctest.Run group. This results in, at the very best, quite confusing behavior. I think we would want to make it possible to detect when running within a group, and have testing.T.Run panic in this case.

My experimental implementation of the synctest package includes a synctest.Sleep function by necessity: It was much easier to implement with an explicit call to advance the fake clock. However, I found in writing these tests that I often want to sleep and then wait for any timers to finish executing before continuing.

I think, therefore, that we should have one additional convenience function:

package synctest

// Sleep pauses the current goroutine for the duration d,
// and then blocks until every goroutine in the current group is idle.
// It is identical to calling time.Sleep(d) followed by Wait.
//
// The caller of Sleep must be in a goroutine created by Run,
// or a goroutine transitively started by Run.
// If it is not, Sleep panics.
func Sleep(d time.Duration) {
	time.Sleep(d)
	Wait()
}

The net/http package was not designed to support testing with a fake clock. This has served as an obstacle to improving the state of the package's tests, many of which are slow, flaky, or both.

Converting net/http to be testable with my experimental version of synctest required a small number of minor changes. A runtime-supported synctest would have required no changes at all to net/http itself.

Converting net/http tests to use synctest required adding an in-memory net.Conn. (I didn't attempt to use net.Pipe, because its fully-synchronous behavior tends to cause problems in tests.) Aside from this, the changes required were very small.

My experiment is in https://go.dev/cl/587657.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc

Commenting here due to @rsc's request:

Relative to my proposal #65336, I have the following concerns:

Goroutine grouping: the only precedent for goroutine having a user-visible identity is runtime.LockOSThread(), and even then, it is set-only: a goroutine can not know whether it is locked to a thread or not without parsing runtime.Stack() output. Having these special "test mode" goroutines feels like a violation of goroutines being interchangeable anonymous workers, insofar as the Go runtime hides the goroutine ID from user code. Having a global wait is acceptable in the case of tests since it is unlikely for background goroutines to be present to interfere with the wait (and possibly actually desirable to catch those too).
Overriding standard library behavior: again, there is no precedent for standard library functions to behave differently based on what goroutine they are called from. The standard idiomatic way to do this is to define an interface (e.g. fs.FS) and direct all calls through the interface, and the implementation of the interface can be mocked at test time. If it is desirable to keep the current Run()/Wait() API, I would still strongly advocate for not changing the behavior of the standard time package, and instead incorporate a mock clock implementation in another package (likely under testing).

Regarding overriding the time package vs. providing a testing implementation:

The time package provides a number of commonly-used, exported functions, where code that makes direct use of these functions cannot be properly tested. I think this makes it unique in the standard library. For example, code which directly calls time.Sleep cannot be tested properly, because inserting a real delay inherently makes a test slow, and because there is no non-flaky way for a test to ensure that a certain amount of time has elapsed.

In contrast, we can test code which calls os.Open by providing it with the name of a file in a test directory. We can test code which calls net.Listen by listening on a loopback interface. The io/fs.FS interface may be used to create a testable seam in a system, but it isn't required.

Time is fundamentally different in that there is no way to use real time in a test without making the test flaky and slow.

Time is also different from an fs.File or a net.Conn in that there is only one plausible production implementation of time. A fs.FS might be the local filesystem, or an embedded set of static files, or a remote filesystem of some kind. A net.Conn might be a TCP or TLS connection. But it is difficult to come up with occasions outside of tests when time.Sleep should do anything other than sleep for the defined amount of time.

Since we can't use real time in tests, we can insert a testable wrapper around the time package as you propose. This requires that we avoid the idiomatic and easy-to-use time package functions. We essentially put an asterisk next to every existing function in the time package that deals with the system clock saying, "don't actually use this, or at least not in code you intend to test".

In addition, if we define a standard testable wrapper around the clock, we are essentially declaring that all public packages which deal with time should provide a way to plumb in a clock. (Some packages do this already, of course; crypto/tls.Config.Time is an example in std).

That's an option, of course. But it would be a very large change to the Go ecosystem as a whole.

runtime: print blocking status of bubbled goroutines in stacks

runtime: disallow closing bubbled chans from outside bubble

testing/synctest: add Test

encoding/json: avoid supurious synctest deadlock detection

runtime: record synctest bubble ownership in hchan

replace time.now linkname

runtime, testing/synctest: verify cleanups/finalizers run outside bub…

runtime, testing/synctest: breaking bubble isolation with Cond is fatal

Change https://go.dev/cl/680695 mentions this issue: internal/fetch: remove goexperiment.synctest

internal/fetch: remove goexperiment.synctest

fix: proper time adjustment to fix flaky TestEtcdManualBackup

Change https://go.dev/cl/701935 mentions this issue: main.star: stop setting GOEXPERIMENT=synctest in x/net builders

main.star: drop GOEXPERIMENT=synctest from Go 1.25+ x/net builders

@neild

Hi @neild, thanks for your excellent blog post here: https://go.dev/blog/synctest , which is still the top Google result for "synctest".

I'd mention though that it uses Run(f func()) throughout, but actually that's the function in internal/synctest. Gophers using synctest actually access it via testing/synctest's Test(t *testing.T, f func(*testing.T)).

It's obviously not a huge deal, but if you've got a little time to fix the blog post, some people or their agents may save on some confusion in the future 😄.

https://pkg.go.dev/internal/synctest#Run
https://pkg.go.dev/testing/synctest#Test

Navigation Menu

Repository navigation

testing/synctest: new package for testing concurrent code #67434

232 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

testing/synctest: new package for testing concurrent code #67434

Description

Activity

aclements commented on May 16, 2024

neild commented on May 16, 2024

apparentlymart commented on May 17, 2024

neild commented on May 17, 2024

gh2o commented on May 18, 2024

neild commented on May 18, 2024

neild commented on May 22, 2024

rsc commented on May 23, 2024

gh2o commented on May 29, 2024

neild commented on May 29, 2024

232 remaining items

gopherbot commented on Jun 10, 2025

gopherbot commented on Sep 8, 2025

srabraham commented on Mar 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions