Monthly Archives: July 2016

Kubernetes at Box: All About Being Describable

Sam Ghods, writing on the Box Blog:

This is one key difference between Kubernetes and other orchestration solutions – while most solutions would require you to go to many different systems to manage these pieces (or to write your own glue to tie those systems together), Kubernetes believes that your infrastructure should fundamentally be describable through a set of Kubernetes objects

This might be the best description of the Kubernetes ethos I’ve seen yet, and it’s been my driving force in implementing Kubernetes. I love the idea of describable infrastructure, and I love the idea of disposable infrastructure. Kubernetes gives us both.

Git Protocol Behavior Change At Github.com

We recently saw practically all of our CI tests begin failing if certain tasks attempted to pull private repositories from Github. Specifically, if our Ruby Gemfiles or Go dependencies referenced private repositories, builds would fail with authentication failures.

These errors generally look like this when running bundle install for our project:

Fetching git://github.com/redacted-org/redacted-repo.git
fatal: remote error:
  Repository not found.
Retrying git clone 'git://github.com/redacted-org/redacted-repo.git' "/var/lib/gems/2.1.0/cache/bundler/git/redacted-repo-401a...807f" --bare --no-hardlinks --quiet due to error (2/4): Bundler::Source::Git::GitCommandError Git error: command `git clone 'git://github.com/redacted-org/redacted-repo.git' "/var/lib/gems/2.1.0/cache/bundler/git/redacted-repo-401a...807f" --bare --no-hardlinks --quiet` in directory /tmp has failed.
fatal: remote error:
  Repository not found.

A similar error occurred building a Go project.

The commonality here is that Bundler and godep both recorded the dependencies’ Github URLs as Git protocol. It should be noted that the Git protocol (git://) is unauthenticated and is a pull-only protocol from Github.

Previously, however, these URLs did actually work for authenticated pulls. We were pulling authenticated repositories as late as the evening of July 20, 2016. With no changes to our Ruby environment, our git client, Dockerfile, Gemfile, or Gemfile.lock, these Bundler installs began failing with this error on the 21st.

As a result of this, we engaged in a vicious search-and-destroy to add the following line to all of our build scripts and Dockerfiles:

git config --global url."git@github.com:".insteadOf "git://github.com/"

My working assumption is that, when presented with an attempt to pull a private repository over the Git protocol, Github was silently transforming these pulls into SSH protocol pulls. Sometime overnight between July 20 and July 21, they stopped performing this redirect.

[Update 2016/07/25]: Stacey @ Github Support responded with the following:

We did implement a change recently that corrected our logic to adhere to the correct git protocol.

This means URLs must start with git@github.com.

Thanks for confirming the change, Stacey.