cmd/go: rethink "missing go.mod" mode

@bcmills

A few weeks ago, shortly after GO111MODULE=on became the default, I got to watch Rob using it. One problem he had was that when he worked in preexisting GOPATH directories, or just directories newly created for the purpose of testing one bug or another, there was no go.mod, so every go command reresolved "latest" for any imported dependencies. This made things much slower than they are intended to be (that is, much slower than they are when there is a go.mod).

I think we probably need to rethink what we do without any go.mod. There needs to be some place for the go command to write down its decisions from one command to the next. When people say "modules are slow" I wonder if this case is one of the things they mean.

/cc @bcmills @jayconrod @ianthehat

Some ideas (separated for emoji-response tracking):

When we need to implicitly resolve a package as part of a go build or go test (rather than, say, a go get), we could pull in the latest version found in the module cache instead of trying to get the absolute latest upstream version.

We could revive the “default go.mod” idea (from cmd/go: allow "go get" when outside a module in module mode #24250 (comment) and others).

The downside of that approach is that it would be really easy for users to unintentionally get stuck on some really old version of a dependency and never realize that they need to upgrade it. (That's one of the more annoying failure modes of GOPATH today.)

We could be less willing to run commands outside of a module in general; for example, we could refuse to run go get unless the user passed an explicit “global install” flag (cmd/go: offer a consistent "global install" command #30515), and we could refuse to resolve any dependencies from outside the standard library during a go build.

That would at least prompt the user to run go mod init earlier in their workflow, which would mitigate some of the sources of slowness.

An important use case for no go.mod file are tools that do not use any imported dependencies. For certain software, such as, say, a Go module proxy, or a simple command line tool, the programmers might want to avoid a chicken-and-egg bootstrap problem and therefore depend only on the Go standard library. Whatever the solution, I'd like this use case to be taken into consideration.

@beoran

@beoran, maintaining a go.mod file for a package with few (or no) external dependencies should be a negligible amount of overhead (since it never needs to change), and the go.mod file has some other important effects (such as the go directive, which declares the expected version of the Go language specification).

I think the bug-testing and legacy-GOPATH use cases are more compelling.

Looking at cmd/go/testdata/script/mod_outside.txt, the only "slow success" cases (where there's no go.mod so every execution has to re-resolve unknown imports) are things like 'go run x.go' (or go build, go vet etc) where x.go contains an import of a package from outside the standard library. If we make that case fail with a message about not having a go.mod, then I think we can consider this issue fixed as far as blocking setting GO111MODULE=on by default.

Change https://golang.org/cl/198778 mentions this issue: cmd/go: forbid resolving import to modules when outside of a module

@bcmills

I wrote this comment on the CL:

With this CL applied, I can no longer go run my tools outside a module:

$ go run gioui.org/cmd/gogio
package gioui.org/cmd/gogio: cannot look up module for package "gioui.org/cmd/gogio": working directory is not part of a module

I can't even run (demo) programs:

$ go run gioui.org/apps/hello
package gioui.org/apps/hello: cannot look up module for package "gioui.org/apps/hello": working directory is not part of a module

This behaviour change seems to be a regression, in particular in light of https://golang.org/issue/33518.

"go get" is exempt from the new behaviour, why not "go run"?

@bcmills replied

go get is exempt because people generally only run it once: you run go get, and that resolves dependencies and installs a binary that you then invoke directly.

go run is not exempt because it is more frequently run many times in succession. That imposes a recurring cost (and non-reproducible behavior) to re-resolve the dependencies, which would become a one-time cost (and reproducible behavior) if the resolution could be recorded in a go.mod file.

Let me defend go run outside a module:

First, this CL is taking away a useful, albeit slow, feature without a replacement. I don't consider the usual workaround "use go get and run the installed command" a good enough replacement for go run. See #33518 for my arguments.

Second, this issue seems to be about running local Go packages and standalone programs. Quoting Russ:

Looking at cmd/go/testdata/script/mod_outside.txt, the only "slow success" cases (where there's no go.mod so every execution has to re-resolve unknown imports) are things like 'go run x.go' (or go build, go vet etc) where x.go contains an import of a package from outside the standard library.

I can live with not allowing go run of local files and packages outside a module. What I'm interesting in is running already published modules, such as

$ go run gioui.org/apps/hello
$ go run gioui.org/cmd/gogio -target android gioui.org/apps/hello

Ironically, both gioui.org/cmd/gogio and gioui.org/apps/hello are themselves modules, even though the go run command is run outside one.

Note that even though I personally don't care much about go running local files and packages outside a module doesn't mean it's a good idea to disallow it. Writing a short Go program and then go running it is a very useful and lightweight way to work. Forcing me to do go mod init example.com just to please go run seems like unnecessary bureaucracy (writing this argument out, I'm curious what Rob would say about this extra requirement in his work). play.golang.org doesn't force me to deal with modules, why should go run?

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default.

In short, I think submitting CL 198778 as is would be a mistake and I would be quite disappointed to see a useful feature go away for efficiency reasons.

Finally, is go run still too slow, as claimed by the Russ in the first post? I believe this issue was filed before GOPROXY became the default.

I ran the following experiment:

$ go version
go version devel +f923c66893 Sat Sep 14 13:08:03 2019 +0200 linux/amd64
$ export GOPATH=$(mktemp -d)
$ time go run gioui.org/cmd/gogio
go: finding gioui.org latest
go: finding gioui.org/cmd latest
go: downloading gioui.org v0.0.0-20191003172637-31e12607d739
go: downloading gioui.org/cmd v0.0.0-20191003172637-31e12607d739
go: extracting gioui.org v0.0.0-20191003172637-31e12607d739
go: extracting gioui.org/cmd v0.0.0-20191003172637-31e12607d739
go: downloading gioui.org v0.0.0-20190930145547-3784ece6dd65
go: extracting gioui.org v0.0.0-20190930145547-3784ece6dd65
go: downloading golang.org/x/image v0.0.0-20190802002840-cff245a6509b
go: downloading golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: extracting golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: extracting golang.org/x/image v0.0.0-20190802002840-cff245a6509b
go: finding golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: finding golang.org/x/image v0.0.0-20190802002840-cff245a6509b
gogio: specify a package
exit status 1

real	0m3,664s
user	0m2,028s
sys	0m0,662s
$ time go run gioui.org/cmd/gogio
go: finding gioui.org latest
go: finding gioui.org/cmd latest
gogio: specify a package
exit status 1

real	0m0,511s
user	0m0,413s
sys	0m0,158s

The first run completed in just under 5 seconds, the second in less than a second.

For comparison, I ran gogio inside the gioui.org/cmd module:

$ time go run ./gogio/
gogio: specify a package
exit status 1

real	0m0,369s
user	0m0,334s
sys	0m0,122s

That is, a difference of ~200ms between inside and outside a module.

Ironically, both gioui.org/cmd/gogio and gioui.org/apps/hello are themselves modules, even though the go run command is run outside one.

That's an interesting input for #30515 / #34506. In particular, it suggests that we may want to allow go run (and possibly go install) to accept the same “global install” flag as go get in order to explicitly opt in to the slower path.

Even then, I think that only get, if that, should resolve dependencies that are not already specified by the module(s) containing the package argument(s).

cmd/go: forbid resolving import to modules when outside of a module

Change https://golang.org/cl/200297 mentions this issue: cmd/go: forbid module pattern 'all' when outside a module

I think we missed a case: go mod verify should probably error out when outside of a module.

We should check test coverage for go mod download too.

Change https://golang.org/cl/202697 mentions this issue: cmd/go: make 'go mod verify' report an error outside a module

Change https://golang.org/cl/205065 mentions this issue: cmd/go: make 'go mod why' require a main module

cmd/go: make 'go mod why' require a main module

Change https://golang.org/cl/216597 mentions this issue: doc/go1.14: go command behavior with GO111MODULE=on outside modules

doc/go1.14: go command behavior with GO111MODULE=on outside modules

Navigation Menu

Repository navigation

cmd/go: rethink "missing go.mod" mode #32027

27 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cmd/go: rethink "missing go.mod" mode #32027

Description

Activity

bcmills commented on May 14, 2019

bcmills commented on May 14, 2019

bcmills commented on May 14, 2019

beoran commented on May 15, 2019

bcmills commented on May 15, 2019

rsc commented on Sep 19, 2019

gopherbot commented on Oct 3, 2019

eliasnaur commented on Oct 4, 2019

eliasnaur commented on Oct 4, 2019

bcmills commented on Oct 4, 2019

27 remaining items

gopherbot commented on Oct 10, 2019

bcmills commented on Oct 21, 2019

gopherbot commented on Oct 22, 2019

gopherbot commented on Nov 4, 2019

gopherbot commented on Jan 27, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions