Repository navigation
spec: allow basic interface types to instantiate comparable type parameters #56548
Description
TL;DR
This is a restatement of proposal #52509 by @zephyrtronium. That issue has collected a lot of comments; starting anew will make it easier for readers to follow along again. The primary additional contributions in this issue are the formulation of the exact spec changes, a corresponding new API for go/types, and a discussion of these changes in the context of a future Go without the current implementation restrictions on interfaces. As a result, this issue is long, but the actually proposed change is small.
Background
The Go 1.18 release introduced the predeclared identifier comparable which denotes the set of all non-interface types that are comparable. More precisely, it is the set of (non-interface) types for which == and != are defined and for which these operations are guaranteed to not panic.
The types in the comparable type set do not correspond to the types of comparable operands: for instance, struct{ f any } is comparable but applying == on the field f may panic.
For clarity, in the following we use the term strictly comparable for the types in comparable, and spec-comparable for types of comparable operands. Strictly comparable types are spec-comparable, but not the other way around. Types that are not spec-comparable are simply not comparable.
Because in Go 1.18 and Go 1.19 constraint satisfaction means interface implementation, not all spec-comparable types satisfy the comparable constraint. The generic map type
type mymap[K comparable, V any] map[K]Vcannot be instantiated with struct{ f any } for K, but the built-in map type permits that. This is a major handicap for the design of generic data structures. See #51257 for another example.
Over the course of this year, many ideas have been proposed to address this problem, while trying to preserve the rule that constraint satisfaction and interface implementation remain the same. Here's a (possibly incomplete) list of proposals for reference: #51338, #52474, #52531, #52614, #52624, #53734.
If we want any to satisfy comparable, then constraint satisfaction can't be the same as interface implementation. A non-comparable type T (say []int) implements any, but T does not implement comparable (T is not in comparable's type set). Therefore any cannot possibly implement comparable (the implements relation is transitive - we cannot change that). So if we want any to satisfy the comparable constraint, then constraint satisfaction can't be the same as interface implementation.
A pre-Go1.18 implementation of the compiler did not treat interface implementation and constraint satisfaction the same, which led to initial confusion because of the resulting inconsistency (see #50646). Eventually the inconsistency was considered an implementation bug (in hindsight this was done without enough consideration of all the implications). With more experience it appears that this inconsistency made it easier to use generic Go with commonly used Go types.
@zephyrtronium's #52509 and this issue propose that we go back to this pre-Go1.18 implementation of constraint satisfaction.
Proposal
We change the spec to use a different rule for constraint satisfaction than for interface implementation: we want spec-comparable types to satisfy comparable; i.e., we want to be able to use any type for which == is defined even if it may not be strictly comparable.
Specifically, in the language spec section on Instantiations, we propose to change the 2nd rule from (old):
After substitution, each type argument must implement the constraint (instantiated, if necessary) of the corresponding type parameter. Otherwise instantiation fails.
to (new):
After substitution, each type argument must satisfy the constraint (instantiated, if necessary) of the corresponding type parameter. Otherwise instantiation fails.
We also add a new paragraph defining what "satisfy" means:
A type
Tsatisfies a constraint interfaceCif
TimplementsC; orCcan be written in the forminterface{ comparable; E }, whereEis a basic interface andTis comparable and implementsE.
With this change, constraint satisfaction matches interface implementation but also contains an exception for spec-comparable types. This exception permits the use of interfaces as type arguments which require strict comparability.
This is the entire proposal.
Observations
-
The proposed change expands the set of types that satisfy a constraint, therefore no existing programs are invalidated and the proposal is fully backward-compatible. See the section on Implementation below for when the feature becomes available.
-
The spec has various implementation restrictions: interfaces that are not basic may only be used as type constraints, or as elements of other interfaces used as constraints. They cannot be the types of values or variables, or components of other, non-interface types. Because of these restrictions, if the type argument
Tis a (non-type parameter) interface, it must be a basic interface. Thus, we can ignore the case of type-constrained interfaces (such asinterface{ ~[]int }) which can only hold non-comparable types. For the same reason, ifTis a non-interface type with an interface field (such asstruct{ f F }whereFis an interface),Fmust be a basic interface. In other words, we don't need to complicate the rules to explicitly disallow such types forT: any interface that is or appears in a (non-type parameter) type argument must be a basic interface and therefore is spec-comparable. -
The spec has additional implementation restrictions in place:
comparableand interfaces containing methods may not appear as terms in a union (with more than one term). Because of these restrictions, ifcomparableappears in an interface (or an embedded element), it can be "factored out to the top"; the same is true for methods. More precisely, constraint interfaces can always be written in one of two forms:interface{ comparable; E }orinterface{ U; E }, whereEstands for all the interface's methods (if any), andUdenotes a union of non-interface types (if any). (An interface of the forminterface{ comparable; U; E }can always be reduced to the forminterface{ U'; E }by eliminating all non-strictly-comparable terms fromU). -
For
interface{ U; E }constraints, constraint satisfaction is the same as interface implementation: nothing changes. For a typeTto satisfy the constraint, it must be in the union and satisfy all the methods, if any. -
For
interface{ comparable; E }constraints, constraint satisfaction is looser: we still expect a type argumentTto implement all the methodsE, but we allow anyTwhich is spec-comparable, even if it is not strictly comparable. This is not statically type-safe and must be remedied with a run-time check for==. -
Because the new rule invalidates static type safety, the definition of strictly comparable is not quite correct anymore:
==on an operand of a strictly comparable type parameter type may in fact panic at run-time. To be able to separate the various kinds of comparability, strictly comparable now more precisely denotes the set of types for which==and!=are defined and for which these operations won't panic if type safety was never broken through the instantiation of a (strictly) comparable type parameter with a spec-comparable type argument. -
If a type parameter
Pis spec-comparable, it is always strictly comparable (but see the previous observation). Therefore, if a type parameter is used as a type argument for a constraint of the forminterface{ comparable; E }, the rule thatPmust "only" be spec-comparable doesn't weaken any static type guarantees becausePwill be strictly comparable. -
The pre-Go 1.18 implementation, before the inconsistency documented in spec: document/explain which interfaces implement
comparable#50646 was reported, matched the proposed rules.
Eventually we may want to remove the existing implementation restrictions. This will also require a generalization of the rule for constraint satisfaction. To ensure backward compatibility, the proposed rule here should match a future, more general rule adjusted for the current implementation restrictions. But at the very least, the proposed rule should not permit more programs now than might be permitted with a more general rule for constraint satisfaction. See the appendix for an attempt at such a generalization.
Examples
Currently, any does not implement the comparable constraint. With the proposed change any will be permitted as type argument for comparable: comparable can be written as interface{ comparable; E } and thus the new rule applies, and any is spec-comparable and implements E (where E is the empty interface in this case).
Currently, the type parameter P in the type parameter list
[P interface{ comparable; fmt.Stringer }]cannot be instantiated with the type S
type S struct {
data any
}
func (S) String() stringbecause S is not strictly comparable. With the proposed change, S must only be spec-comparable (which it is) and implement fmt.Stringer (which it does).
More examples
type mymap[K comparable, V any] map[K]V
func _[P any, Q comparable]() {
var (
_ map[func()]string // invalid with current and new rules: func() is not comparable
_ map[struct{ f P }]string // invalid with current and new rules: P is not comparable
_ map[struct{ f Q }]string // valid: key type is strictly comparable
_ mymap[any, string] // currently invalid, valid with new rules: any supports ==
_ mymap[struct{ f any }, string] // currently invalid, valid with new rules: type of f supports ==
_ mymap[struct{ f func() }, string] // invalid with current and new rules: f is not comparable
_ mymap[struct{ f int }, string] // valid: key type is strictly comparable
_ mymap[struct{ f P }, string] // invalid with current and new rules: P is not comparable
_ mymap[struct{ f Q }, string] // valid: key type is strictly comparable
)
}Implementation
The proposal requires minor changes to the typechecker:
- We have already implemented this alternative semantics at tip so that we can experiment. The new compiler flag
-altcomparableenables the new behavior (disabled by default):
go tool compile -altcomparable <file.go> // applied to a single file
go build -gcflags=-altcomparable <pkg path> // applied to a package
etc.
If the proposal is accepted for Go version 1.nn, we will replace this flag with a version check that enables the new behavior starting with Go 1.nn.
- Clients of go/types need to be able to check for constraint satisfaction, which is now slightly different from interface implementation. We cannot add a flag to the existing
types.Implementsfunction without breaking backward-compatibility. Instead, we propose to add the following exported function:
// Satisfies reports whether type V satisfies the constraint interface T.
//
// The behavior of Satisfies is unspecified if V is Typ[Invalid] or an uninstantiated
// generic type.
func Satisfies(V Type, T *Interface) bool- We may also want to provide a variant of
types.Comparablethat implements the notion of strict comparability. We can wait with this until a clear need arises.
All these changes are trivial (the internal implementations exist, they just need to be exported).
If this proposal is accepted in time, it could become effective with the Go 1.20 release.
Related spec issues
- The current definition of the type set of the predeclared type
comparableis incorrect in the spec. The definition is as follows:
The predeclared interface type
comparabledenotes the set of all non-interface types that are comparable. Specifically, a typeTimplements comparable if:
Tis not an interface type andTsupports the operations==and!=; orTis an interface type and each type inT's type set implements comparable.
The first rule (T supports the operations == and !=) also includes types (such as structs) which may not be strictly comparable (see the structs used in the examples above). The rule needs to be amended to exclude types for which == may panic.
- The definition of comparable operands in the spec is missing a dedicated rule for operands of type parameter type. Without it, type parameters (whose underlying types are their constraint interfaces) fall into the category of spec-comparable interface values. This is incorrect. A type parameter-specific rule needs to be added to this section.
Both these spec corrections should be made irrespective of whether this proposal is accepted or not.
Appendix
In a hypothetical future version of Go ("FutureGo") where generalized interfaces may be used as ordinary types, the constraint satisfaction rules must be suitably extended and expressed in terms of type sets. @ianlancetaylor proposes the following FutureGo rules:
Given a (constraint) interface
C, we define two type setsS1andS2as follows:
S1is the type set ofCcomputed in the usual way, except that wherevercomparableappears inC
it is treated as the set of all (spec-)comparable types (not just the strictly comparable types); andS2is the type set ofCcomputed in the usual way, except that wherevercomparableappears inC
it is treated as the set of all types,any.
Given these special type sets S1 and S2, we can define general constraint satisfaction as follows:
A type
Tsatisfies a constraint interfaceCif
Tis not an interface type, andTis an element of type setS1; orTis a (non-type parameter) interface type, the type set ofTis a subset of the type setS2,
and the intersection of the type set ofTwith the type setS1is not empty; orTis a type parameter andTimplementsC.
(The case for type parameters can be folded into the case of non-interface types, but for the following discussion it is clearer to handle it separately.)
If T is not an interface (or type parameter) type, it is not hard to see why the respective sub-rule makes sense: instead of only accepting strictly comparable types for comparable, the use of the type set S1 permits any spec-comparable type.
If T is an interface type, we also want to accept it as a type argument for comparable: more precisely, any interface type, irrespective of its type set, supports == and thus is spec-comparable (we could say that non-basic interfaces don't support == by default, but that's an separate discussion). Effectively this means that comparable is the same as any in this case, hence the type set S2. However, we (probably) want to exclude interface types with type sets for which we can prove that == can't possibly succeed. For instance, interface{ ~[]int } is spec-comparable (all interfaces support ==, but see the comment on that above), but its type set only contains non-comparable types: comparing operands of type interface{ ~[]int } will always panic. The intersection of the type set of such interfaces with the type set S1 will be empty (if it were not empty, the interface would contain at least one spec-comparable type and so == might always happen to succeed in a concrete application). This explains the second part of the rule for interfaces.
Finally, if T is a type parameter, nothing changes and we can simply ignore this case below.
To make sure we remain forward-compatible with FutureGo, i.e., that these generalized constraint satisfaction rules do not restrict our current, limited proposal, we need to show that our current proposal doesn't permit more types for a constraint than the FutureGo rules.
We can do this by applying the current implementation restrictions to the FutureGo constraint satisfaction rules. With those restrictions in place, from the Observations section we know that constraint interfaces can appear in only two (canonical) forms:
a) interface{ comparable; E }; or
b) interface{ U; E }
where E stands for all the methods in the constraint (if any), and U denotes a union of non-interface types (if any). Therefore, the respective type set S1 is either
S1a which is the type set of interface{ comparable; E } with comparable denoting all spec-comparable types; or
S1b which is the type set of interface{ U; E } (unchanged).
Similarly, the type set S2 is either
S2a which is the type set of interface{ E }; or
S2b which is the type set of interface{ U; E } (unchanged).
Now we can look at the type argument T and how the generalized rules apply:
If T is not an interface type, per the generalized rules, T must be an element of S1. Therefore, T must either be an element of S1a, or S1b. If the former is true, T must be spec-comparable and implement the methods E - this corresponds to the our proposed exception for interface satisfaction. If the latter is true, T is simply an element of the type set of the constraint, which means T implements the constraint. This corresponds to the non-exceptional part of the proposed rule for constraint satisfaction.
If T is a (non-type parameter) interface type, with the implementation restrictions in place, T must be a basic interface. Per the generalized rules, the type set of T must be a subset of S2a and the intersection of T's type set and S1a must not be empty, or the type set of T must be a subset of S2b and the intersection of T's type set and S1b must not be empty. If the former is true, T implements the methods E, and because T is a basic interface it is also spec-comparable. This corresponds to the proposed exception for constraint satisfaction. If the latter is true, T simply implements the constraint, which again corresponds to the non-exceptional part of the proposed rule for constraint satisfaction.
In summary, if we apply the current implementation restrictions to FutureGo, the FutureFo constraint satisfaction rules reduce to the rules proposed in this issue. This provides some insurance that the proposed rules won't cause problems in FutureGo.
Activity
zephyrtronium commentedon Nov 3, 2022
This seems like a small and clever rule with intentionally broad consequences. I'm not sure why this approach is preferable to, in particular, #52509 (comment) by @Merovius. Adding the "satisfies" relation seems to be helpful for the go/types API, but we can do that and also have a list of small explicit rules rather than a very big implicit one. Perhaps that's just my reading.
griesemer commentedon Nov 3, 2022
The approach suggested by @Merovius changes the meaning of type sets to include interfaces, which seems like a pretty big change. The end result may still be the same, but I haven't tried to prove that in any way.
If we want an interface, say
anyto satisfycomparable- which thecomparableissues are all about - constraint satisfaction simply cannot be the same as interface implementation. Thus, the simplest change I can think of is to separate the two and introduce an exception for exactly the cases which we want to cover. Unless I'm overlooking something (which is entirely possible), it seems that the proposed rule does exactly what developers want: to be able to use a spec-comparable type where we request acomparabletype argument, similar to what we always could do with map keys.nightlyone commentedon Nov 3, 2022
Since after implementing this comparable is equivalent to any since any will satisfy the comparable constraint, I don't understand why comparable is needed. What can we model with a comparable that will include any that we can't model with any alone?
There may be advantages when stenciling out the actual usages, but type safety is not improved over just using any.
Also when following this proposal one again cannot model comparable types without special support for equality that comparable enables right now. I don't see a plan for regaining this capability in in the proposal.
griesemer commentedon Nov 3, 2022
comparableis not the same asanywith this proposal nor do we state that anywhere. It's not possible to use a non-comparable type (such as[]int) as type argument for a type parameter constrained bycomparable- yet if the constraint wereany, it would be possible. What this proposal does is carving out just the right (I think) exception to permit types that in Go are considered comparable as type arguments constrained bycomparable, even if those types are not strictly comparable (==may panic). We retain all the static safety elsewhere. Static type safety is definitely stronger than withany.I do not understand what you mean by your second paragraph. Can you provide an example of the capability you're missing?
Finally, you may want to try and play with the actual implementation of this feature which is available to you at tip (see the Implementation section of the proposal). Perhaps that helps clarifying the impact this would have on your code.
apparentlymart commentedon Nov 4, 2022
Thanks for sharing this proposal. It broadly makes sense to me and I believe it will allow me to write the programs I described wanting to write in some of the previous proposal discussions (which I will not repeat here).
I do have a clarifying question, though:
The proposed text constrains
Eininterface{ comparable; E }to be a Basic interface. The spec currently defines that as an interface whose body consists only of methods.The following section of the specification defines "Embedded interfaces" without describing the interaction between "Basic interface" and "Embedded interface". Is it true to say that an interface which embeds a basic interface is itself a basic interface as long as it otherwise meets the definition of a basic interface?
Concretely, is
Bin the following example a "basic interface" for the sake of satisfying the proposed new rule?griesemer commentedon Nov 4, 2022
Yes,
Bin your example would be basic interface. Per the spec:In other words, we mean all interfaces that can be written in the form
interface{ comparable; M1; M2; ... }(where theMi's are methods) after "inlining" or "flattening" of any embeddings.apparentlymart commentedon Nov 4, 2022
Ahh yes, thank you for clarifying that. I clumsily read "can be defined entirely by" as "are defined entirely by".
atdiar commentedon Nov 4, 2022
Thank you for the detailed proposal.
Sorry in advance for the long-winded comment. I think this proposal is a step toward a solution although there are a few things that I think differ from the conclusions of #52509 and #53734 and these differences may be important.
(also, note that #53734 and its predecessor suggests to differentiate between interface implementation and constraint satisfaction)
To be honest, I've found some of the explanations a bit hard to follow personally (which might be from my own limitations).
I think that the issue is that:
comparablestill seems to be treated as a special interface (which seems to be confusing others apparently)(5) is about struct types whose fields may be interfaces. https://go.dev/play/p/JkAcqvXlirF
In current Go, these types are accepted as map keys even though they might trigger a runtime panic. So they can be used in Set datastructures that are built upon maps. I think this was one use-case that was mentioned by people.
They are indeed satisfying the
comparableconstraint if we define such constraint as the proposition: has the comparison operators == and !=.That however does not mean that these types would implement a potential future
comparableinterface. Admittedly, that part has been underdiscussed but seems that any composite type would implementcomparableif and only if every part of that type implementscomparable.For a struct, that would be every field. For an array, the type argument to the array constructor should implement
comparable.Following this, we would not require a distinction between spec and strictly comparable. That takes care of issue 1 and issue 5 as the distinction would merely be between implementing and satisfying.
But now, what about 2,3,4 ? What is even implementing an interface and satisfying a constraint, why does it differ? How to build a simple mental model?
That's where I think there has been attempts to establish the (long) list of requirements for each in the spec but I think it shows that the mental model may need to be clarified so that it is made simpler and more obvious.
If we take a step back and think about it (I will probably be paraphrasing #53734):
comparableis just an interface defined by the set of constraints { has == and != operators}Usatisfies a constraint interfaceViffUbelongs to the type set ofVby definitionIimplements an interfaceJiff we can show that all types that satisfyIsatisfyJ. (i.e. the type set defined by I is a subset of the type set defined by J i.e. the set of constraints definingJis a subset of the set of constraints definingI)So as can be seen (in A,D,E), the definition of constraint satisfaction is the basis for a definition of interface implementation instead of the reverse.
We have only one kind of type sets and one kind of comparability. That's simpler.
Also because assignment to an interface value must be true for all satisfying typed values, it is by definition using interface implementation as a requirement. In the case of
comparable, it would correspond to what is defined as strict comparability in the current proposal.Constraint statisfaction is what is used by type parameters, so
should still be able to instantiate generic map keys, just like in normal Go code, since it has the comparison operators.
Now on the question of whether interface types belong to type sets or not?
... it depends.
We just need to apply the constraint rules rulesto interface types.
Obviously that means that interface types may belong to type sets. Sometimes they would not, depending on the type set. e.g. interface{ int | string}
Just like regular types.
Another question is whether all interface types should satisfy comparable ?
As mentioned in this proposal, it might seem less clear for
interface{ ~[]int}.I would still tend to say yes but it is perhaps not mandatory. Having these operators doesn't seem to rely on type sets as seen with
any. Quite the opposite, the availability of these operators determine the type set instead.In any case, it should be evident that it does not implement
comparable, however. There is a difference between a type and the elements of the set it denotes.griesemer commentedon Nov 4, 2022
I believe the mental model needed here remains very simple: Constraint satisfaction is the same as interface implementation (nothing changes), but we introduce an exception so that Go's (spec-) comparable types can also be used where (strictly)
comparabletype arguments are required.That is all, and that is exactly what users are asking for. The reason we need this is that historically, Go was not statically type-safe with respect to
==: the operation can panic in some cases. But when writing generic code, we want to preserve type-safety as much as possible (which is whycomparableis not the same as spec-comparable), this is simply something users don't want to give up. These two goals are conflicting, and the only way to bridge the gap is through an exception. Which is what we propose.Now, to answer your questions:
We have always had (spec-)comparable and we cannot change that or it would break backward-compatibility. With Go 1.18 we introduced
comparablewhich stands for strictly comparable types. We could makecomparablethe same as spec-comparable (a sort of late fix for a "bug" in the generics design), one could even get rid ofcomparable(make it meanany), see e.g. proposal: type parameters are comparable unless they exclude comparable types #52614. But this and similar ideas were vehemently rejected repeatedly. People wantcomparableto provide some guarantees, which is what we have now. Thereforecomparablemust be different from spec-comparable.Constraint satisfaction depends mostly (not partially!) on interface implementation because we want to keep it that way. We just added an exception for spec-comparable types. That is the whole point of this proposal.
comparableis a special interface: it is predeclared because we don't have another mechanism in the language to denote strictly comparable types (and we don't want to invent one either, a predeclared identifier works fine). Again, if we accept 1),comparablemust be different from spec-comparable.FutureGo is a thought experiment. We want to make sure that this proposal holds up even if we were to remove the current implementation restrictions. Or, looking the other way, we want to make sure that we can remove the current implementation restrictions and not run into problems because of this proposal. To that effect, @ianlancetaylor has proposed a set of (plausible) rules for constraint satisfaction that we believe would work with such a future Go, and then we show that this proposal doesn't conflict with those hypothetical future rules. It is possible that these future rules can be expressed more clearly and concisely. All this is relevant here only insofar that it gives us some confidence that this proposal is reasonable. If it feels confusing to you, just ignore the appendix.
The built-in go
maptype behaves exactly the same in all cases: the key type must be spec-comparable. If you define your own generic map type with a key type parameter that iscomparable, it also accepts spec-comparable types. With this proposal, your example typecan be used to instantiate a generic map key. Again, that is the goal of this proposal.
I very much agree with you that it would all be much simpler if there was only one type of comparability - and we have tried that before (again see #52614 and perhaps others). But users want stronger guarantees.
To summarize:
comparablereally means strictly comparable (but see the next point)anyfor a strictlycomparabletype parameter (andanyis not strictly comparable)(Regarding your last two questions: a) should interfaces be parts of type sets, and b) should all interfaces satisfy comparable. For a): probably not - but we don't need to answer this question for this proposal. For b) the type
interface{ ~[]int }cannot be used as an ordinary type, so the question is moot for that case. A type parameter constrained by this interface is and will not be (strictly)comparable, nor is it spec-comparable. This won't change and is I think what you'd expect.)I recommend you play with the implementation available at tip. Setting the compiler flag
-altcomparablewill give you the proposed behavior. Thanks.atdiar commentedon Nov 4, 2022
@griesemer thanks for the reply. I realize that I misread the examples and made a mistake. As you correctly state, the struct example works. So that's one less issue.
Regarding
comparablehowever, if we distinguish between interface implementation and constraint satisfaction, do we really still need to have two notions of comparability?We should be able to keep compile-time safety (strict comparability) since it uses interface implementation and interface types would still satisfy the
comparableconstraint. Just not implement it.Basically instantiation requires constraint satisfaction. Assignability requires interface implementation (which forces strict comparability) ?
Interface implementation being constraint satisfaction of the members of the type set of a type instead of constraint satisfaction of the type itself.
47 remaining items
zigo101 commentedon Dec 8, 2022
Is there an open issue which proposes to use
comparableas value types?atdiar commentedon Dec 8, 2022
@go101 see #51338
Edit: oh right... "open" 😅.. Not that I know of...
zigo101 commentedon Dec 8, 2022
I found this issue from #51338, which has been closed. ;D
ianlancetaylor commentedon Dec 9, 2022
@go101 I don't know of one. Personally I think that permitting variables of type
comparableis similar to permitting variables of all non-basic interface types. I don't see a reason to do justcomparable.gopherbot commentedon Dec 13, 2022
Change https://go.dev/cl/457095 mentions this issue:
spec: introduce notion of strict comparabilitygopherbot commentedon Dec 13, 2022
Change https://go.dev/cl/457437 mentions this issue:
spec: describe new semantics for comparable and constraint satisfactionspec: introduce notion of strict comparability
spec: describe new semantics for comparable and constraint satisfaction
griesemer commentedon Dec 14, 2022
This is now implemented and documented. Closing.