SBT—Scala’s almighty build tool—continues to amaze me, both, with the amount of WTFs delivered to me, and with the sheer power it puts in my hands. Sometimes I feel like I don’t understand what’s going on at all, and then I find myself writing SBT plugins to automate complex tasks across our build and test infrastructure in a breeze. I curse at SBT when I have to explain how SBT works to my peers, and I love SBT because more than other build tools I know it helps me automate all aspects of project life cycles, even non-trivial tasks in proprietary infrastructures.
At my company we run about 30 services written in Scala, on top of Twitter’s Finagle framework. Our CI infrastructure continuously builds, tests, and deploys each service after every single git push—we can get changes to production in less than 15 minutes. As part of each service deployment pipeline we run regression tests, ie, all integration tests of all services that depend on the service, against a special canary instance of the deployed service. We use these regression tests to catch accidental incompatibilities in service ABIs, and behaviour regressions across services.
While the idea worked well for us, our actual implementation had a few WTFs on its own. We built fat JARs of integration tests, shoved them to HDFS and had years-old Bash script to download these fat JARs in regression test jobs. A rather intricate and complex process whose details kept obstructing progress in our test infrastructure—even just updating Scala Test in the integration tests of a service was hard work, let alone switching to another test framework or upgrading our Scala version. A couple of days ago it got in our way again, and instead of hacking around in this mess again, we chose to get rid of all this and move to SBT for this job, so I wrote an SBT plugin to add a list of dependent services to a service and define a task to
- ask our CI server for the Git commit hashes running in production of all dependent services
- clone or fetch the Git repositories of these services, in parallel,
- checkout the “production commit” in each repository,
- and run
sbt it:testin each repository, in parallel, with some special parameters to select the canary instance of this service.
To my surprise this took me less than three hours, and I just needed three attempts to get the plugin to work on our CI infrastructure.
I’ll write another post about the code soon, but first I’d like to say my heartfelt thanks to all SBT contributors and SBT plugin authors that work hard to put this power into my hands.
SBT, we surely got a love-hate relationship, but you keep amazing me!