cmd/go: exclude vendor dir from matching ...

There are some closed issues about this, but I'd like to bring it up again.

Any time you're building an application and you make an update to any package in the repo (of which there are often many), you run go test ./... to make sure you haven't broken your program. This worked fine before vendoring became a thing, but with vendoring, it runs all the tests of vendored code as well. While you may want to do that once in a while, such as when you update the code in the vendor directory, it's generally not something you want to do every time your code changes (because it shouldn't affect code you depend on at all).

I'd like to suggest that ... ignore the vendor directory by default. I almost never want to have any command touching the vendor directory directly - not go vet, not go fmt, not go build, not go test, not go install, not go fix.

If the community feels that there is utility in being able to match the vendor directory, I suggest adding a flag called -vendor (or whatever) that indicates ... should match the vendor directory. But by default, I think almost everyone would prefer it not match the vendor directory.

As we start down the path of having an official package management solution, I think vendoring will get more and more common.... and this will become more and more of a pain.

(and yes, I know you can do the go test $(go list | grep -v vendor) thing, but that's shell and OS-dependent, and ugly, and just bad UX)

I feel similarly about this issue.

vendor could be ignored when walking its parent, so ./... would ignore it but ./vendor/... wouldn't.

I would like to add the votes of 21 people that do not want to have their editor run go test against vendor/ at all https://youtrack.jetbrains.com/issue/GO-2366 and 19 others that don't want gofmt to run against vendor/ as well https://youtrack.jetbrains.com/issue/GO-2412. One could argue that the editor should be smarter and send a list of packages to the tool but then why should users be forced to either have to use $(go list | grep -v vendor), which mind you is not possible on Windows, or use an editor to achieve this? WSL does not count as it introduces yet again more work for the user, thus bad UX.

Also, please notice tools like glide, one popular dependency management tool, where there's even a dedicated command to list packages and omit the vendor directory in order to make life easier.

This is clearly an UX problem that the users should not have to face.

Previous discussion:

cmd/go: exclude vendor directories from wildcard matches? #11659 exclude vendor directories from wildcard matches?
cmd/go: go test ./... fails on go 1.6 because dependencies are tested #14417 cmd/go: go test ./... fails on go 1.6 because dependencies are tested
cmd/go: go generate ./... will recurse into vendor #17497 cmd/go: go generate ./... will recurse into vendor
Exclude vendor/ from linters #17879 Exclude vendor/ from linters

Marking as Go 1.9 for further consideration and decision.

I would also like to add that this is what a Windows user currently has to write when using Windows CMD to work around the vendor issue

for /f "" %G in ('go list ./... ^| find /i /v "/vendor/"') do @go test %G

which is a lot worse than go test $(go list | grep -v vendor).

@natefinch

@natefinch thanks for raising this again.

Regarding the usual argument that "./... means all sub packages should avoid making vendor a especial case":

vendor is an especial case.
./... already ignores folders with names starting with . or _, so there's a precedent supporting this proposal.

Are there good reasons why the vendor directory cannot simply be renamed _vendor? Kind regards, *Paul Reiber*http://about.me/reiber

…

On Tue, Feb 14, 2017 at 2:53 PM, Ernesto Jiménez ***@***.***> wrote: @natefinch <https://github.com/natefinch> thanks for raising this again. Regarding the usual argument that "./... means all sub packages should avoid making vendor a especial case": 1. vendor *is* an especial case. 2. ./... already ignores folders with names starting with . or _, so there's a precedent supporting this proposal. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#19090 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADlQB4xORqJ5Z9Y1dbkOTD-C4dlEPLxks5rcjBWgaJpZM4MBALL> .

@PaulReiber

@PaulReiber see this comment: #11659 (comment)

"vendor" is not being renamed. That ship has long sailed.

@ianlancetaylor can you please add #11659 to the list of related issues as hopefully your comment will be more visible than mine. Thank you

@dlsniper

@dlsniper Done, thanks.

This was overwhelmingly mentioned in the tools survey I conducted last year. Combining all feedback I got (individuals, mailing list questions, StackOverflow, etc), this particular problem was one of the top problems that has been mentioned.

This has be discussed at length and rejected. I think the real solution is #11193. What people really don't want is to run unneeded tests. With proper test caching, it doesn't matter go test ./... tests vendored packages or not.

That doesn't help for all the rest of the tooling. I don't want go vet complaining about problems in vendored code. I definitely do not want go fmt or go fix changing things in vendored code. I don't want go install installing binaries that happen to be in vendored repos.

The root of it is this: without vendoring, ... never matches dependencies that arent't in your repo, because they're not subdirectories. I just want to maintain that functionality while vendoring.

Then don't invoke go fmt or go fix on ./..., which is very problematic even if there aren't any vendored packages. The core problem is people tend to overuse the ./... pattern. The vendor problem is just a red herring.

This hasn't been true for over a decade. Any supported Windows variant includes a shell that, while somewhat different from Unix shells, is very well capable of such things. I wouldn't be surprised if it even provides a grep command by default. It certainly does so for many other common Unix programs (generally as an alias for the native Powershell command with roughly equivalent functionality).

@slrz

@slrz So with your suggestion windows users will now have to install a shell they're not familiar with, learn that shell, rewrite their scripts in it, and this is easier than you running go test ./... ./vendor/... until a complete vendoring solution is found?

@slrz

@slrz if you could find something simpler than:

for /f "" %G in ('go list ./... ^| find /i /v "/vendor/"') do @go test %G

I'd be happy to hear about it.

Meanwhile, this feature was requested by the Go users, which proved with a fairly good technical evidence that not matching vendor/ in ... is desired or at least have a way to disable that behavior.

I believe that the current way that's implemented offers a good balance of that while still allowing vendor/ to be matched if so needed.

It will rarely be so that you need to type: ./... ./vendor/... ./.../vendor/... let alone ./... ./vendor/... ./.../vendor/... ./.../vendor/.../vendor/... (vendor flattening should be a thing by now in any package manager).

Most likely you'd need to type: ./... ./vendor/... which is still just one character longer than --no-vendor ./... and one shorter than --skip-vendor ./... but it's definitely easier to type / remember than the current way to do things.

Also, special cases do exist today, afaik testdata is one such case (if not only).

So please, try to understand that there are some of us who are really happy to see this, and per the commend here: #19090 (comment) that number might actually be bigger than you think. So a quality of life improvement for so many people while still allowing for the current behavior to exist is a welcomed change imho.

I hope this helps seeing "the other side" as well.

@sgen

@sgen Heck, we're talking about programmers here. They will figure out something to exclude items containing a constant substring from a list.

Assuming they even want that particular quirky behaviour instead of the one you'd expect from reading the description of patterns in the help text, without the part about this special case.

@slrz

@slrz this isn't the quirk. The quirk is that vendor is a special directory for the go build tool. This completes the implementation of the quirk.

We were in a bad state previously when vendor was incompletely implemented. We were in the worst position of all - a quirk that was inconsistent.

If purity doesn't convince you, practicality should: In my experience teaching Go and being a local expert in a large-ish company on Go tools, this was an almost universal stumbling block when people went to lint or test their code. Nobody knew to use go list and grep ahead of time, and most of them wrinkled their nose and made some disparaging comment about go's tools.

@dcheney-atlassian

@dcheney-atlassian

But it does have to work on Windows, which are the largest population of Go users, and have neither shell aliases or nice things like grep, or substitution.

Can you please cite some source of Windows Go users being "the largest population of Go users"?

Meanwhile, let me refer to a survey of 3,595 respondents:

When asked which operating systems they develop Go on, 63% of respondents say they use Linux, 44% use MacOS, and 19% use Windows, with multiple choices allowed and 49% of respondents developing on multiple systems. The 51% of responses choosing a single system split into 29% on Linux, 17% on MacOS, 5% on Windows, and 0.2% on other systems..

The windows version is the most popular download of windows/dl. This was reported to me in private correspondence a year or so ago, I don't have access to verify this fact directly.

@rsc

Out of respect for everyone, I think we should stop the bike shedding now and maybe open up an issue if we find a problem in the current implementation as asked by @rsc.

Finally, please note that comments like Heck, we're talking about programmers here. are really not in the spirit of cooperation / Go. Yes, we are all programmers, but that doesn't mean we all share the same opinion.

The fact that vendor/ is special is undeniable and if users really really care about traversing that, then maybe they should have to use special scripts instead of those who don't (I've borrowed this last bit from someone who said it in a private discussion).

Lets not have the team lock down the discussion as I consider that to be a bit unfriendly. Thank you.

But I want to paint it chartreuse!

@dlsniper

@dlsniper

if you could find something simpler than:
for /f "" %G in ('go list ./... ^| find /i /v "/vendor/"') do @go test %G
I'd be happy to hear about it.

The following should work in a default Windows system, having installed only the Go MSI package:

go test $(go list ./... | sls -NotMatch '/vendor/')

The sls ("select string") seems to be the Powershell equivalent to grep.

My soln till this mess gets sorted is with npm package.json. Something like the following:

{
  "name": "myapp",
  "version": "1.0.0",
  "scripts": {
    "test:win": "for /f \"\" %G in ('go list ./... ^| find /i /v \"/vendor/\"') do @go test -v %G",
    "test:unix": "go test -v $(go list ./... | grep -v /vendor/)",

    "bench:unix": "go test -bench=. -v $(go list ./... | grep -v /vendor/)",
    "bench:win": "for /f \"\" %G in ('go list ./... ^| find /i /v \"/vendor/\"') do @go test -v -bench=. %G",

    "start:unix": "go run *.go",
    "start:win": "for /f \"\" %G in ('dir /b *.go') do @go run %G"
  }
}

To test on windows npm run test:win, to bench on windows npm run bench:win, and to run on windows npm start:win

To test on linux/osx npm run test:unix, to bench on linux/osx npm run bench:unix, and to run on linux/osx npm start:unix

better late than never!

Update all deps and add vendor directory

Navigation Menu

Repository navigation

cmd/go: exclude vendor dir from matching `...` #19090

57 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cmd/go: exclude vendor dir from matching ... #19090

Description

Activity

mvdan commented on Feb 14, 2017

dlsniper commented on Feb 14, 2017

ianlancetaylor commented on Feb 14, 2017

dlsniper commented on Feb 14, 2017

ernesto-jimenez commented on Feb 14, 2017

PaulReiber commented on Feb 14, 2017

dlsniper commented on Feb 14, 2017

ianlancetaylor commented on Feb 14, 2017

rakyll commented on Feb 14, 2017

minux commented on Feb 15, 2017

natefinch commented on Feb 15, 2017

minux commented on Feb 15, 2017

57 remaining items

slrz commented on Mar 30, 2017

sgen commented on Mar 30, 2017

dlsniper commented on Mar 30, 2017

slrz commented on Mar 30, 2017

spenczar commented on Mar 30, 2017

cznic commented on Mar 30, 2017

dcheney-atlassian commented on Mar 30, 2017

dlsniper commented on Mar 31, 2017

sgen commented on Mar 31, 2017

slrz commented on Mar 31, 2017

AhmetS commented on Apr 24, 2017

mantzas commented on May 19, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

cmd/go: exclude vendor dir from matching `...` #19090