Sunday, 18 August 2013

Biting the Bullet(s) - Bpipe, Git and Vagrant (and Chef/Puppet)

I'm a big fan of Bpipe. Why I like it can wait for another post, and the homepage makes a good case. I'd relied on it in a Torque/PBS environment that recently moved to SLURM, which Bpipe does not currently support. Self-interest and open source code meant that if it was going to get added, I could at least get the ball rolling by contributing (or attempting to).

I had dabbled in some source code changes to Bpipe previously on a VM that I've not fired up in a few months. I don't quite recall how I installed it, the various path changes, and issues. The code changes I made are not under source control - they were just sitting there, and I never submitted them.

I know enough to know that's not the way to do it. But there are lots of things I know I should have set up and be using, including source control on everything and having reproducible development environment for contributing to a project such as Bpipe.

I've started on them, there are rough notes for all but a well documented coherent process for them ... <insert excuse here>.

But my reliance on Bpipe for being able to quickly reuse components, call parallel tasks and overall use supercomputing resources with minimal effort has forced my hand. It is clear that:

  1. I need to have Bpipe support SLURM.
  2. I need to setup a new development environment to work on Bpipe.  
  3. I need a way to have track the code changes I make, and share them with the community. 
  4. I need to get to work!
I've been wanting to try Vagrant ( for a while, as well as try out some provisioning software like Chef and Puppet. My GitHb account has been setup but dormant for a while. All that has been lacking is my will to set aside some time to do it.

With any luck, the following list will turn into links to the posts I make as I progress through each of these steps.
  1. Vagrant (and Chef/Puppet)
  2. Git
  3. Bpipe & SLURM