Repository navigation
cmd/go: module graph pruning and lazy module loading #36460
Description
I propose to change cmd/go to avoid loading transitive module dependencies
that have no observable effect on the packages to be built.
The key insights that lead to this approach are:
-
If no package in a given dependency module is ever (even transitively)
imported by any package loaded by an invocation of thegocommand, then an
incompatibility between any package in that dependency and any other package
has no observable effect in the resulting program(s). Therefore, we can
safely ignore the (transitive) requirements of any module that does not
contribute any package to the build. -
We can use the explicit requirements of the main module as a coarse filter
on the set of modules relevant to the main module and to previous
invocations of thegocommand.
Based on those insights, I propose to change the go command to retain more
transitive dependencies in go.mod files and to avoid loading go.mod files
for “irrelevant” modules, while still maintaining high reproducibility for build
and test operations.
Design doc: http://golang.org/design/36460-lazy-module-loading.
Activity
gopherbot commentedon Feb 3, 2020
Change https://golang.org/cl/217557 mentions this issue:
cmd/go: add a regression test for package import cycles guarded by build tagsgopherbot commentedon Feb 19, 2020
Change https://golang.org/cl/220080 mentions this issue:
design/36460: add design for lazy module loadingbcmills commentedon Feb 19, 2020
A detailed design is in CL 220080.
(You can view the rendered Markdown by opening the
.mdfile at the latest patch set and clicking thegitileslink.)ChrisHines commentedon Feb 20, 2020
👍 I think that is a nice property to have. I have some projects for which
go mod vendoris sane, butgo mod tidyupdates thego.modfile in undesirable ways due to test-of-test dependencies. If I understand this proposal correctly that problem would go away.bcmills commentedon Feb 20, 2020
I added a few examples (in our usual
cmd/goscript test format) to the design doc.@ChrisHines, the third example explicitly covers the behavior for test-of-test dependencies.
bcmills commentedon Feb 21, 2020
CC @julieqiu since this may affect how we display dependencies on
pkg.go.dev.CC @heschik, @dmitshur because this may affect the package search algorithms used by
goimportsandgodoc.CC @bep because I seem to recall that Hugo is doing something unusual with module dependencies independent of Go package imports. (But note that we are unlikely to actively support the use of Go modules for purposes other than compiling Go source code.)
bep commentedon Feb 21, 2020
Our unusual usage is fairly well anchored in the "Go project". That said, I have read your design document twice, and I think this is a good thing (even for our unusual use) -- but it would be easier to determine with a prototype (I find this topic to be a little bit of a mind twister and I don't fully understand the examples in the document).
288 remaining items
cmd/go/internal/modload: remove go117EnableLazyLoading
cmd/go/internal/modload: use "pruned" instead of "lazy" to describe p…
go.mod: upgrade to go 1.17
go.mod: upgrade to go 1.17
_content/doc/modules/pruning: add a placeholder redirecting to “Manag…
_content/ref/mod: document Go 1.17 lazy loading and module graph pruning
go.mod: upgrade to go 1.17
go.mod: upgrade to go 1.17
doc/go1.17: clarify Modules changes