Tuesday, 22 October 2013

Well that went quickly...

What seems like only last week but was almost two months ago was a post on what I intended to do given I needed Bpipe working under SLURM. 

Long story short - things went well: https://code.google.com/p/bpipe/source/browse/ReleaseNotes.txt

I'd intended to document the whole process as I went along, but as often is the case, necessity dictates that getting the job done means sometimes neglecting to take the time to record how you did it.

Not that I am too regretful. Diving headfirst has been a great refresher in version control and a chance to try Git, as well as try some rudimentary environments using Vagrant and Puppet

I don't think that the saying 'Show, don't tell' really applies to blog posts, but in order to get another post out the door, I'll risk it.

My steps in betting a Bpipe SLURM implementation were going to involve (the show):
  1. Vagrant (and Chef/Puppet)
  2. Git
  3. Bpipe & SLURM
The tell is:
  1. Vagrant, Puppet and some nasty shell scripts to have a reproducible dev environment for Bpipe:  https://github.com/lonsbio/bpipe-vagrant/tree/master
  2. See 1 and 3.
  3. Using 1, clone and develop a version of Bpipe that (sort of) works under SLURM! https://code.google.com/r/andrewlonsdale-bpipe-dev/
There is still work to do - it is preliminary support only - including adding MPI and SMP support for Bpipe SLURM. This will likely require more work on this (https://github.com/lonsbio/slurm-cluster-vagrant).

I won't promise to document these next steps, and then forget to as I go head first into doing them

However with any luck there will be some more 'tell' soon.

Sunday, 18 August 2013

Biting the Bullet(s) - Bpipe, Git and Vagrant (and Chef/Puppet)

I'm a big fan of Bpipe. Why I like it can wait for another post, and the homepage makes a good case. I'd relied on it in a Torque/PBS environment that recently moved to SLURM, which Bpipe does not currently support. Self-interest and open source code meant that if it was going to get added, I could at least get the ball rolling by contributing (or attempting to).

I had dabbled in some source code changes to Bpipe previously on a VM that I've not fired up in a few months. I don't quite recall how I installed it, the various path changes, and issues. The code changes I made are not under source control - they were just sitting there, and I never submitted them.

I know enough to know that's not the way to do it. But there are lots of things I know I should have set up and be using, including source control on everything and having reproducible development environment for contributing to a project such as Bpipe.

I've started on them, there are rough notes for all but a well documented coherent process for them ... <insert excuse here>.

But my reliance on Bpipe for being able to quickly reuse components, call parallel tasks and overall use supercomputing resources with minimal effort has forced my hand. It is clear that:

  1. I need to have Bpipe support SLURM.
  2. I need to setup a new development environment to work on Bpipe.  
  3. I need a way to have track the code changes I make, and share them with the community. 
  4. I need to get to work!
I've been wanting to try Vagrant (http://www.vagrantup.com/) for a while, as well as try out some provisioning software like Chef and Puppet. My GitHb account has been setup but dormant for a while. All that has been lacking is my will to set aside some time to do it.

With any luck, the following list will turn into links to the posts I make as I progress through each of these steps.
  1. Vagrant (and Chef/Puppet)
  2. Git
  3. Bpipe & SLURM

Monday, 17 June 2013

First Post (The Post That Hurts The Most)

The first post, as is well known, is the post that hurts the most.

I find the first post hard because there is a temptation to try and make it a perfect representation of what the blog aims to be: full of interesting content, written for large audience and a valuable resource to others in the field - with a few jokes as well.

But since none of those things are true (yet), all I can say is that this will simply be a bioinformatics blog with tips, tricks, challenges and (hopefully) solutions that I come across as a bioinformatician.

My background prior to bioinformatics was in software; I learn more about biology every day, and many posts will be about biological that I learn that I find new or exciting. They may be quite pedestrian to those with a deeper knowledge, but one of the best things about being a bioinfomatican is the exposure to such a wide variety of biological knowledge and I.d like to share that as I go.

All views expressed are my own etc etc, not those of my employers or groups I am affiliated with etc etc.