Rewrite docs on godep
This commit is contained in:
Tim Hockin
parent
b02edb9d2b
commit
42db070c3f
|
|
@ -152,24 +152,11 @@ images.
|
|||
|
||||
#### Dependency management
|
||||
|
||||
Kubernetes build/test scripts use [`godep`](https://github.com/tools/godep) to
|
||||
manage dependencies.
|
||||
|
||||
```sh
|
||||
go get -u github.com/tools/godep
|
||||
```
|
||||
|
||||
The Godep version that Kubernetes is using is listed in `Godep/Godep.json` (in
|
||||
the kubernetes repo root). See what version you are running with this command:
|
||||
|
||||
```sh
|
||||
godep version
|
||||
```
|
||||
|
||||
Developers planning to manage dependencies in the `vendor/` tree may want to
|
||||
explore alternative environment setups. See [using godep to manage
|
||||
dependencies](godep.md).
|
||||
Kubernetes uses [`godep`](https://github.com/tools/godep) to manage
|
||||
dependencies.
|
||||
|
||||
Developers who need to manage dependencies in the `vendor/` tree should read
|
||||
the docs on [using godep to manage dependencies](godep.md).
|
||||
|
||||
|
||||
## Build with Bazel/Gazel
|
||||
|
|
@ -178,7 +165,6 @@ Building with Bazel is currently experimental. For more information,
|
|||
see [Build with Bazel].
|
||||
|
||||
|
||||
|
||||
## Workflow
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,128 +1,181 @@
|
|||
# Using godep to manage dependencies
|
||||
|
||||
This document is intended to show a way for managing `vendor/` tree dependencies
|
||||
in Kubernetes. If you are not planning on managing `vendor` dependencies go here
|
||||
[Godep dependency management](development.md#godep-dependency-management).
|
||||
in Kubernetes. If you do not need to manage vendored dependencies, you probably
|
||||
do not need to read this.
|
||||
|
||||
## Alternate GOPATH for installing and using godep
|
||||
## Background
|
||||
|
||||
There are many ways to build and host Go binaries. Here is one way to get
|
||||
utilities like `godep` installed:
|
||||
As a tool, `godep` leaves much to be desired. It builds on `go get`, and adds
|
||||
the ability to pin dependencies to exact git version. The `go get` tool itself
|
||||
doesn't have any concept of versions, and tends to blow up if it finds a git
|
||||
repo synced to anything but `master`, but that is exactly the state that
|
||||
`godep` leaves repos. This is a recipe for frustration when people try to use
|
||||
the tools.
|
||||
|
||||
Create a new GOPATH just for your go tools and install godep:
|
||||
This doc will focus on predictability and reproducibility.
|
||||
|
||||
## Theory of operation
|
||||
|
||||
The `go` toolchain assumes a global workspace that hosts all of your Go code.
|
||||
|
||||
The `godep` tool operates by first "restoring" dependencies into your `$GOPATH`.
|
||||
This reads the `Godeps.json` file, downloads all of the dependencies from the
|
||||
internet, and syncs them to the specified revisions. You can then make
|
||||
changes - sync to different revisions or edit Kubernetes code to use new
|
||||
dependencies (and satisfy them with `go get`). When ready, you tell `godep` to
|
||||
"save" everything, which it does by walking the Kubernetes code, finding all
|
||||
required dependencies, copying them from `$GOPATH` into the `vendor/` directory,
|
||||
and rewriting `Godeps.json`.
|
||||
|
||||
This does not work well, when combined with a global Go workspace. Instead, we
|
||||
will set up a private workspace for this process.
|
||||
|
||||
The Kubernetes build process uses this same technique, and offers a tool called
|
||||
`run-in-gopath.sh` which sets up and switches to a local, private workspace,
|
||||
including setting up `$GOPATH` and `$PATH`. If you wrap commands with this
|
||||
tool, they will use the private workspace, which will not conflict with other
|
||||
projects and is easily cleaned up and recreated.
|
||||
|
||||
To see this in action, you can run an interactive shell in this environment:
|
||||
|
||||
```sh
|
||||
export GOPATH=$HOME/go-tools
|
||||
mkdir -p $GOPATH
|
||||
go get -u github.com/tools/godep
|
||||
# Run a shell, but don't run your own shell initializations.
|
||||
hack/run-in-gopath.sh bash --norc --noprofile
|
||||
```
|
||||
|
||||
Add this $GOPATH/bin to your path. Typically you'd add this to your ~/.profile:
|
||||
## Restoring deps
|
||||
|
||||
To extract and download dependencies into `$GOPATH` we provide a script:
|
||||
`hack/godep-restore.sh`. If you run this tool, it will restore into your own
|
||||
`$GOPATH`. If you wrap it in `run-in-gopath.sh` it will restore into your
|
||||
`_output/` directory.
|
||||
|
||||
```sh
|
||||
export GOPATH=$HOME/go-tools
|
||||
export PATH=$PATH:$GOPATH/bin
|
||||
hack/run-in-gopath.sh hack/godep-restore.sh
|
||||
```
|
||||
|
||||
## Using godep
|
||||
This script will try to optimize what it needs to download, and if it seems the
|
||||
dependencies are all present already, it will return very quickly.
|
||||
|
||||
Here's a quick walkthrough of one way to use godeps to add or update a
|
||||
Kubernetes dependency into `vendor/`. For more details, please see the
|
||||
instructions in [godep's documentation](https://github.com/tools/godep).
|
||||
If there's every any doubt about the correctness of your dependencies, you can
|
||||
simply `make clean` or `rm -rf _output`, and run it again.
|
||||
|
||||
1) Devote a directory to this endeavor:
|
||||
Now you should have a clean copy of all of the Kubernetes dependencies.
|
||||
|
||||
_Devoting a separate directory is not strictly required, but it is helpful to
|
||||
separate dependency updates from other changes._
|
||||
## Making changes
|
||||
|
||||
The most common things people need to do with deps are add and update them.
|
||||
These are similar but different.
|
||||
|
||||
### Adding a dep
|
||||
|
||||
For the sake of examples, consider that we have discovered a wonderful Go
|
||||
library at `example.com/go/frob`. The first thing you need to do is get that
|
||||
code into your workspace:
|
||||
|
||||
```sh
|
||||
export KPATH=$HOME/code/kubernetes
|
||||
mkdir -p $KPATH/src/k8s.io
|
||||
cd $KPATH/src/k8s.io
|
||||
git clone https://github.com/$YOUR_GITHUB_USERNAME/kubernetes.git # assumes your fork is 'kubernetes'
|
||||
# Or copy your existing local repo here. IMPORTANT: making a symlink doesn't work.
|
||||
hack/run-in-gopath.sh go get -d example.com/go/frob
|
||||
```
|
||||
|
||||
2) Set up your GOPATH.
|
||||
This will fetch, but not compile (omit the `-d` if you want to compile it now),
|
||||
the library into your private `$GOPATH`. It will pull whatever the default
|
||||
revision of that library is, typically the `master` branch for git repositories.
|
||||
If this is not the revision you need, you can change it, for example to
|
||||
`v1.0.0`:
|
||||
|
||||
```sh
|
||||
# This will *not* let your local builds see packages that exist elsewhere on your system.
|
||||
export GOPATH=$KPATH
|
||||
hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v1.0.0'
|
||||
```
|
||||
|
||||
3) Populate your new GOPATH.
|
||||
Now that the code is present, you can start to use it in Kubernetes code.
|
||||
Because it is in your private workspace's `$GOPATH`, it might not be part of
|
||||
your own `$GOPATH`, so tools like `goimports` might not find it. This is an
|
||||
unfortunate side-effect of this process. You can either add the whole private
|
||||
workspace to your own `$GOPATH` or you can `go get` the library into your own
|
||||
`$GOPATH` until it is properly vendored into Kubernetes.
|
||||
|
||||
Another possible complication is a dep that uses `gopdep` itself. In that case,
|
||||
you need to restore its dependencies, too:
|
||||
|
||||
```sh
|
||||
cd $KPATH/src/k8s.io/kubernetes
|
||||
./hack/godep-restore.sh
|
||||
hack/run-in-gopath.sh bash -c 'cd $GOPATH/src/example.com/go/frob && godep restore'
|
||||
```
|
||||
|
||||
4) Next, you can either add a new dependency or update an existing one.
|
||||
|
||||
To add a new dependency is simple (if a bit slow):
|
||||
If the transitive deps collide with Kubernetes deps, you may have to manually
|
||||
resolve things. This is where the ability to run a shell in this environment
|
||||
comes in handy:
|
||||
|
||||
```sh
|
||||
cd $KPATH/src/k8s.io/kubernetes
|
||||
DEP=example.com/path/to/dependency
|
||||
godep get $DEP/...
|
||||
# Now change code in Kubernetes to use the dependency.
|
||||
./hack/godep-save.sh
|
||||
hack/run-in-gopath.sh bash --norc --noprofile
|
||||
```
|
||||
|
||||
To update an existing dependency is a bit more complicated. Godep has an
|
||||
`update` command, but none of us can figure out how to actually make it work.
|
||||
Instead, this procedure seems to work reliably:
|
||||
### Updating a dep
|
||||
|
||||
Sometimes we already have a dep, but the version of it is wrong. Because of the
|
||||
way that `godep` and `go get` interact (badly) it's generally easiest to hit it
|
||||
with a big hammer:
|
||||
|
||||
```sh
|
||||
cd $KPATH/src/k8s.io/kubernetes
|
||||
DEP=example.com/path/to/dependency
|
||||
# NB: For the next step, $DEP is assumed be the repo root. If it is actually a
|
||||
# subdir of the repo, use the repo root here. This is required to keep godep
|
||||
# from getting angry because `godep restore` left the tree in a "detached head"
|
||||
# state.
|
||||
rm -rf $KPATH/src/$DEP # repo root
|
||||
godep get $DEP/...
|
||||
# Change code in Kubernetes, if necessary.
|
||||
rm -rf Godeps
|
||||
rm -rf vendor
|
||||
./hack/godep-save.sh
|
||||
# Regenerate removed BUILD, licenses.
|
||||
touch vendor/BUILD
|
||||
./hack/update-bazel.sh
|
||||
./hack/update-godep-licenses.sh
|
||||
# If you haven't followed this doc step-by-step and haven't created a dedicated GOPATH,
|
||||
# make sure there is no client-go or other staging repo in $GOPATH before running the next command.
|
||||
./hack/update-staging-godeps.sh
|
||||
git checkout -- $(git status -s | grep "^ D" | awk '{print $2}' | grep ^Godeps)
|
||||
hack/run-in-gopath.sh bash -c 'rm -rf $GOPATH/src/example.com/go/frob'
|
||||
hack/run-in-gopath.sh go get -d example.com/go/frob
|
||||
hack/run-in-gopath.sh bash -c 'git -C $GOPATH/src/example.com/go/frob checkout v2.0.0'
|
||||
```
|
||||
|
||||
_If `go get -u path/to/dependency` fails with compilation errors, instead try
|
||||
`go get -d -u path/to/dependency` to fetch the dependencies without compiling
|
||||
them. This is unusual, but has been observed._
|
||||
This will remove the code, re-fetch it, and sync to your desired version.
|
||||
|
||||
### Removing a dep
|
||||
|
||||
This happens almost for free. If you edit Kubernetes code and remove the last
|
||||
use of a given dependency, you only need to restore and save the deps, and the
|
||||
`godep` tool will figure out that you don't need that dep any more:
|
||||
|
||||
## Saving deps
|
||||
|
||||
Now that you have made your changes - adding, updating, or removing the use of a
|
||||
dep - you need to rebuild the dependency database and make changes to the
|
||||
`vendor/` directory.
|
||||
|
||||
```sh
|
||||
hack/run-in-gopath.sh hack/godep-save.sh
|
||||
```
|
||||
|
||||
This will run through all of the primary targets for the Kubernetes project,
|
||||
calculate which deps are needed, and rebuild the database. It will also
|
||||
regenerate other metadata files which the project needs, such as BUILD files and
|
||||
the LICENSE database.
|
||||
|
||||
After all of this is done, `git status` should show you what files have been
|
||||
modified and added/removed. Make sure to `git add` and `git rm` them. It is
|
||||
commonly advised to make one `git commit` which includes just the dependency
|
||||
update and Godeps files, and another `git commit` that includes changes to
|
||||
Kubernetes code to use the new/updated dependency. These commits can go into a
|
||||
single pull request.
|
||||
modified and added/removed. Make sure to sanity-check them with `git diff`, and
|
||||
to `git add` and `git rm` them, as needed. It is commonly advised to make one
|
||||
`git commit` which includes just the dependencies and Godeps files, and
|
||||
another `git commit` that includes changes to Kubernetes code to use (or stop
|
||||
using) the new/updated/removed dependency. These commits can go into a single
|
||||
pull request.
|
||||
|
||||
5) Before sending your PR, it's a good idea to sanity check that your
|
||||
Godeps.json file and the contents of `vendor/ `are ok by running `hack/verify-godeps.sh`
|
||||
## Sanity checking
|
||||
|
||||
_If `hack/verify-godeps.sh` fails after a `godep update`, it is possible that a
|
||||
transitive dependency was added or removed but not updated by godeps. It then
|
||||
may be necessary to perform a `hack/godep-save.sh` to pick up the transitive
|
||||
dependency changes._
|
||||
Before sending your PR, it's a good idea to sanity check that your
|
||||
Godeps.json file and the contents of `vendor/ `are ok:
|
||||
|
||||
It is sometimes expedient to manually fix the /Godeps/Godeps.json file to
|
||||
minimize the changes. However without great care this can lead to failures
|
||||
with `hack/verify-godeps.sh`. This must pass for every PR.
|
||||
```sh
|
||||
hack/run-in-gopath.sh hack/verify-godeps.sh
|
||||
```
|
||||
|
||||
6) If you updated the Godeps, please also update `Godeps/LICENSES` by running
|
||||
`hack/update-godep-licenses.sh`.
|
||||
All this script will do is a restore, followed by a save, and then look for
|
||||
changes. If you followed the above instructions, it should be clean. If it is
|
||||
not, you get to figure out why.
|
||||
|
||||
## Manual updates
|
||||
|
||||
It is sometimes expedient to manually fix the `Godeps.json` file to
|
||||
minimize the changes. However, without great care this can lead to failures
|
||||
with the verifier scripts. The kubernetes codebase does "interesting things"
|
||||
with symlinks between `vendor/` and `staging/` to allow multiple Go import
|
||||
paths to coexist in the same git repo.
|
||||
|
||||
The verifiers, including `hack/verify-godeps.sh` *must* pass for every pull
|
||||
request.
|
||||
|
||||
|
||||
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
|
||||
|
|
|
|||
Loading…
Reference in New Issue