"Those who don't know history are doomed to repeat it." ― Edmund Burke
There are many variations on this saying but the essential element is that it's important, and I would argue useful, to know the history of something. It can provide depth, understanding, and insight.
At the moment there are many debates going on about package management in Go. There are questions being asked, such as how should it work or whose ideas should we follow?
To give context to these ideas it's worth looking at the history of dependency management in Go. It's a story of people, discovery, differences, disconnects, and even a little drama.
In The Beginning...
go getGOPATHGOPATHGOPATHgo get-ucpThere were no versions. It was built around how Google handles source and dependencies. Go is a Google owned and run project so a system built around their style is where this all started.
Godep
GOPATHGOPATHGOPATHThere were some other tools lurking around about the same time but none of them gained the traction of godep.
The Vendor Directory
After awhile there were several tools trying to solve dependency management. A common theme for these projects was the desire for different applications to use different versions of the same dependency.
vendorGOPATHGOROOTvendorvendorDependency Management Like Everyone Else
While godep was popular, some other tools came along that received a significant amount of usage. For example, there was a project I worked on named glide. The idea with Glide was to provide package management similar to how it worked in JavaScript, Python, Rust, PHP, and pretty much all of the languages with modern package managers.
Glide didn't stand alone. There were numerous package managers that had market share. Things were getting splintered.
From Many To One
How can we solve a problem of too many package managers? Form a committee, right?
A committee was formed to figure out package dependency management for Go. Andrew Gerrand, from the Go team at Google, was a member of this committee. There was a second advisory group, which I was a member of, that provided them with insight.
To support the committee a couple of other activities kicked off.
- A survey of the Go community about package management. This asked questions about Go, package management, and package managers. It even queried people on their experience with package management in other languages.
- Interviews with people at companies who used Go. The idea was to understand their needs if they were to build applications around Go.
The committee came to some conclusions on what we needed based on the information from the community and some debate. They wrote up what was needed in a specification.
dep was built to have one new tool that met the needs outlined in the spec. Some of us who worked on tools like godep and glide put our support behind this effort.
vendorThe expectation was that dep, a community initiative with the support of the Go team, would become the standard manager.
But, this is not the end of the story.
Enter vgo
At the GopherCon in Denver in 2017 there was a contributor summit. It was a place where some people could get together to discuss Go the day prior to GopherCon.
At this summit there was time set aside for people to discuss dependency management and we did for some time. Around the conclusion of that time Russ Cox, the current lead of the Go team, came to the table. He made a comment that he could do better if he went off on his own and built something. That something was later announced as vgo. It is the thing he went off on his own, apart from the community, and created.
Despite community objections, vgo was recently accepted as the path forward.
My 2 Cents
This is where we stand now. It's interesting because of the social dynamics.
dep and the spec that started it was based on the community learning its own needs and collaborating on a solution. vgo was built when the Go lead at Google went off on his own to build a solution.
Practical issues with vgo were discussed, at length, and most of them were dismissed. A common reason for that was two assumptions of the Go team:
- The community of package developers will need to change how they do things from today
- Packages will always maintain SemVer compatibility
For the first point to come to reality in large projects like Kubernetes or Docker there will be a significant amount of work. The dependencies they use will need to make changes or be replaced and the way they currently do dependency management will need to change. There's potential for a lot of work.
When there's a lot of work at stake around a refactor it raises questions about cost, loss of velocity, and priority.
Is the second point, about SemVer, possible? Many language package management systems have been using SemVer for years and breaking from SemVer, sometimes by accident, happens. People aren't perfect and disagree on how to do things.
The biggest take away may be the gap between the Go team at Google and the community. The community's solution was rejected. The Go team created a solution in isolation from the community and mostly rejected the community feedback on it. This feedback came from people who weren't just members of the community but experts in complex projects and, in some cases, dependency management.
At the heart of this may be two things. Where the community sits relative to the language and toolchain and the way Google does things different from the majority, for example a monorepo vs multi-repo.
I want to give a special thanks to Matt Butcher who reviewed and helped me work through this post.