We run a lot of background tasks where we don’t necessarily care where the code actually runs: it needs to have Internet access and access to a couple of S3 buckets, and that’s about it. Some of the jobs are appropriate for Hadoop and Spark, which we crank out through AWS ElasticMapReduce, but some is more nuanced.
To begin with, we back everything with etcd. We use this CloudFormation to bootstrap an etcd cluster according to the CoreOS team’s recommendation for production workloads. We only run etcd on these machines, everyone else just looks to them for configuration data, and these machines do not run any other services.
The operations tier of our stack provides essential services to the cluster: a Docker registry, Shipyard, and the Vulcand proxy. CloudFormation is used to load this cluster, after which we load fleet units to provide those services.
Vulcand’s documentation is lackluster, to say the least. While the docs give examples of how to set up your front and back ends with etcdctl, I’ve found that for some operations, you’re best using either the REST API or
vctl. The Docker registry runs on its host on port 5000 and Shipyard on 8080.
I set up a hostname for the vulcand host,
vulcan.int.derr.me, and then set up
shipyard.into.derr.me as CNAMEs.
# ssh to the machine running vulcand, and then enter the vulcan1 container. $ fleetctl ssh -A -unit=vulcand.service coreos-host$ docker exec -ti vulcan1 bash # setup backends vulcan1$ vctl backend upsert -id registry vulcan1$ vctl backend upsert -id shipyard # setup servers behind those backends vulcan1$ vctl server upsert -b registry -id registry1 -url http://<<registry host IP>>:5000 vulcan1$ vctl server upsert -b shipyard -id shipyard1 -url http://<<shipyard host IP>>:8080 # setup frontends for each vulcan1$ vctl frontend upsert -id registry -b registry -route 'Host(`registry.int.derr.me`) && PathRegexp(`.*`)' vulcan1$ vctl frontend upsert -id shipyard -b shipyard -route 'Host(`shipyard.int.derr.me`) && PathRegexp(`.*`)'
Incoming requests are handled by the matching front end. It fields the request to the attached backend, which is served by one or more servers. If we ever decide we’re pulling enough docker images that we should add another Docker server, we would simply add a second server.
vulcan1$ vctl server upsert -b registry -id registry2 -url http://<<registry2 host IP>>:5000
All other hosts in the cluster are just dumb CoreOS machines running Docker, hooked up to Shipyard.