Juju dependencies managed by dep


#1

Starting with the 2.5 develop branch, Juju core has migrated to use upstream dep as the tool to manage its dependencies. For further reading on the tool itself, see https://golang.github.io/dep/. This will replace our home grown godeps.

As a developer, if you use the existing make targets to update your dependencies when switching branches or pulling new revisions, nothing much changes in respect to your current workflow. The main difference is that the make targets have been renamed. See below for details.

One impact will be the creation of a vendor directory (which is git ignored) in the Juju source tree. This contains a flattened copy of Juju’s dependencies and may be quite large (a few hundred MB).

How to use it

There are 2 artifacts which define the dependencies and their locked revisions:

  • Gopkg.toml
  • Gopkg.lock

These essentially replace the godeps dependencies.tsv file. See the dep docs for more details.

Ensure local copy of upstream dependencies are up to date

Instead of:
make godeps
or
godeps -u dependencies.tsv

do this:
make dep
or
dep ensure -vendor-only

The dep ensure command copies the frozen dependencies as specified in the lock file into the ./vendor folder.

The first time will take longer as everything is downloaded again but after that the flattened dependencies are cached locally and subsequent runs are incremental updates.

Switching branches

If you want to work on older 2.3 or 2.4 branches, as always you need to ensure that the dependencies are updated after switching. With this change, you now also need to remove the vendor directory if it exists. The godeps make target has been updated to do this. Or if you prefer to run godeps -u dependencies.tsv manually, you will also need to remove the vendor directory manually as well.

Add a new dependency

Say there’s a brand new upstream repo for which you want to create a Juju dependency. Add it like so:

dep ensure -add github.com/foo/bar

This will pull the upstream code and ensure that the toml and lock files are updated. These files need to be proposed for landed into Juju the same way that a dependencies.tsv update would be.

Bring in an upstream dependency change

Say there has been a change committed to an upstream dependency and you want a new revision to be used by Juju. As is the case with godeps, you first need to pull this upstream dependency change to update your local copy. Then…

Instead of:
godeps -t ./... > dependencies.tsv

do this:
Copy the sha you want to use and edit the Gopkg.toml file to use this sha.
make rebuild-dependencies

This will run dep ensure -v -no-vendor to rebuild the lock file based on Gopkg.toml. These files need to be proposed for landed into Juju the same way that a dependencies.tsv update would be.

Work on upstream dependencies

Say you want to update some code in an upstream dependency like juju/testing.

The locally cloned copy of the repo will exist in the usual location under $GOPATH. The version used when building Juju is picked up from the ./vendor directory so there’s a few extra steps involved in do the work.

  1. Ensure you have cloned the repo locally.

go get -u github.com/juju/testing

  1. Make whatever changes you want to the source code in this location.

These first 2 steps are the same as what we do when using godeps.

You have two choices on how to test your changes with Juju:

  1. Copy the entire repo to the Juju vendor directory.

cp -r $GOPATH/src/github.com/juju/testing $GOPATH/src/github.com/juju/juju/vendor/github.com/juju/testing

You will also need to prevent make build or make install from overwriting your changes to the vendor directory. Either set the environment variable:

export JUJU_SKIP_DEP=true

or

supply it as an argument to make:

make install JUJU_SKIP_DEP=true

  1. Commit and push your changes to your fork of juju/testing and note the commit sha. Edit the Gopkg.toml file and run make rebuild-dependencies.

Now you can build and test your changes on the juju repo

make install
juju bootstrap...

After you are happy with the changes, you can put up a pull request etc and merge your changes upstream.

After landing the change upstream, use the steps outlined in the previous section to update the toml and lock files and propose the change to the Juju repo.

make rebuild-dependencies
git commit ...
git push ...


#2

Are there plans for the new mechanism in golang 1.11 … ie modules?


#3

Yes! When it is ready. We’re still using 1.10 and 1.11 is not yet GA. Also, modules support is “experimental” even in 1.11. The upstream k8s repos which are the cause of this change recommend dep or godep (but godep is deprecated). Until they are proven to work well with modules, we’ll have to stick with dep.


#4

@wallyworld can we hide the packages output similar to go-install target.

go-build:
	@echo 'go build $$PROJECT_PACKAGES'
	@go build $(PROJECT_PACKAGES)

Should I make a PR?


#5

I’ve encountered errors similar to the following:

grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export github.com/cloud-green/monitoring: unable to update repository: error: object file .git/objects/66/6e3beca3cb4f6b5c8f176ba80daf2b3adbd84e is empty

Indeed, this is a git error, Dep’s local cache was all messed up. The docs did come to the rescue here:

Remove the local cache dir: rm -rf $GOPATH/pkg/dep/sources

The docs do leave readers with the ominous:

There are no known cases where, in the course of normal operations, dep can irreparably corrupt its own local cache...

Such statements are always ominous. I’m not sure how I got myself into the bad state. I don’t recall an abrupt exit of any kind. In any case the fix works!


#6

The packages output has now been hidden.


#7

I believe this has broken our snap building as it’s using the godeps plugin.

https://git.launchpad.net/~juju-qa-bot/+git/juju-edge-snap/tree/snap/snapcraft.yaml#n17

I’ve added a bug and looking if there’s a simple fix we can try.

https://bugs.launchpad.net/juju/+bug/1786815


#8

The first example of “on how to test your changes with Juju” only works if you have JUJU_SKIP_DEP true in your environment variables. Otherwise you’re changes get overwritten.


#9

Sorry Simon, missed this earlier. I’ve landed a fix to make go build less verbose.