Meeting the creator of my favorite automation server

What do you do when you see a tweet announcing that Kohsuke Kawaguchi (the creator of Jenkins CI) is looking to meet in your area with ‘real-world developers outside the Silicon Valley echo chamber’? Do you think twice before replying, or do you instantly say ‘Yes. Let’s meet’! What about the fact that you are not even doing ‘classical’ software development? Can you expect to help Kohsuke regain his confidence that he is in ‘continuous touch’ to Jenkins practitioners’ as in the early days of the project?

I first met Kohsuke Kawaguchi in the summer of 2014 when I had an opportunity to present about the BioUno work using Jenkins at the JUC 2014 meeting in Boston. It was a very busy couple of days and I did not have much time to connect with him or with the CloudBees group that were busy answering Dev Ops questions to more than 400 people that attended that meeting. Now Kohsuke twitted that he was looking to meet with people in the Boston area to strengthen his connection with the OSS Jenkins practitioners. As Kohsuke put it, ‘in the early days of Jenkins before it was called Jenkins, I and my colleagues were using it to deliver our software. That gave me opportunities to deeply understand the problems developers are facing, and that drove much of feature development. Lately I’ve been feeling like I lost that touch to practitioners. Nowadays I only hear about specific tactical problems that some specific users have. That’s good, but those are limited & skewed data points for OSS Jenkins project. Since I was going to (be in) the area, I thought it’d be a good opportunity to meet real-world developers outside (the) Silicon Valley echo chamber. I’m hoping to meet maybe 3 companies and spend 2 hour each. I’d love you to walk me through how a change committed by developer gets to your customers, as if I’m a newly hired engineer. Once I understand how your org works, I’d like to discuss about your pain points, what things go wrong, and what you are thinking about addressing’. All of these were admirable points that Kohsuke was trying to address, the only problem was that I work with Jenkins in a way totally different than the 99% of Jenkins practitioners out there.

Invitation accepted

I was thrilled when Kohsuke twitted back his plans for the visit to Boston, and even more so, when he replied to me ‘I’m still interested’ to my warning that I don’t use Jenkins for its intended use but for creating life/data science application platforms. He would come to my company for a 2 hour meeting at the end of the day. To share my excitement I emailed my BioUno colleague Bruno Kinoshita in New Zealand, and invited my boss named (no kidding) Jeremy Jenkins to attend. Finally things got even better when Kohsuke mentioned that he would be accompanied by Jesse Glick, senior developer from CloudBees.

Preparing for meeting Kohsuke

The most important planning notes for this meeting came from Bruno in a few well-thought out words: ‘Wow, that’s great news! Kohsuke is an amazing person. Send my regards to him, and if you drink with him, pour some beer in the Japanese style kind like this (see picture) . Kohsuke taught us this in a bar, and I have friends in Sao Paulo that are still pouring beer to each other like that". The problem was I’m not a beer-drinker, and we would not be going out with Kohsuke as he was already booked with other plans after our meeting.

So in my ‘dry, cold science’ way, I’ve prepared an outline with links to some live Jenkins projects to demonstrate the ways that we have been using Jenkins in the spirit of the Biouno project, and prepared to discuss future Jenkins improvements that could ease my current ‘pain points’ and could facilitate further adoption of Jenkins in the life and data sciences fields.

The visit

It was late afternoon when Kohsuke and Jesse arrived at the research facility where I work in Cambridge, MA. We started with a visit to the high throughput biology lab where industrial robots set up screening assays for the discovery of new medicines. I knew that this would be interesting for a pair of techno-geeks, but more importantly, I wanted to give them an idea of the type of problems we try to solve in life and data sciences using Jenkins. I think it was effective and caught their attention, although Kohsuke still took quick glimpses into his phone messenger app. For the remaining two hours of the visit, we huddled in a small conference room discussing the various ways we had adapted Jenkins as an image processing and data analysis platform for life sciences. We also discussed the BioUno project and the Active Choices plugin (which was renamed mostly on Jesse’s recommendation from the original UnoChoice name). The interactive interface parameters afforded by the Active Choices plugin so critical for data analytics were a major discussion point. Jesse noted that what we seem to be building is a new application framework based on Jenkins. Finally, we’ve discussed what Jenkins improvements are needed to strengthen its position among other scientific data pipelining and workflow platforms (such as Knime and Galaxy) as I will expand on below.

Can we improve Jenkins OSS for life/data science applications?

Jenkins is a great task integration and workflow execution platform and has proved its value in thousands of continuous integration and continuous delivery projects. However, in the life sciences space Jenkins is practically unknown (not withstanding the efforts of the BioUno project to spread the word about Jenkins) and it has some rather strong entrenched rivals in the face of Knime and Galaxy two data integration/pipelining and workflow orchestration projects. Naturally, applications that are developed with focus on particular fields face challenges when used outside their main domain. As Galaxy or Knime would not fit very naturally in development operations, similarly, Jenkins does not immediately stand out as a natural fit in life/data science applications. Jenkins draws much of its strength from a rich ecosystem of plugins and currently only a handful of life-science and data science oriented plugins exist (all contributed by BioUno). This makes an ‘out of the box’ Jenkins installation impractical for life sciences. But beyond that, there are some cross-cutting improvements that would benefit both the development operations as well as life/data sciences communities. I was glad to hear both Kohsuke and Jesse agree with this. Such improvements revolve around a better global search engine for Jenkins, build metadata, artifact re-use and (what I call) relational builds or perhaps bi-directional build dependencies. It is likely that these improvements will find their way into the OSS version of Jenkins as a result of the current Jenkins community efforts. I also think that the introduction of the Pipeline Plugin a.k.a. ‘pipeline as code’ is a good example of a cross-cutting improvement, as it further increases the build reproducibility and artifact traceability both critical for scientific data flows.

Finally, Jenkins needs better graphical project and pipeline configuration and presentation tools. These features are already present in Knime and Galaxy but mostly missing from Jenkins. Such graphical tools would make Jenkins project and pipeline comprehension a lot easier than it currently is. I hope that others in the community feel similarly about this issue, so some solutions will start to emerge.

Final Thoughts

I’m thankful that the Jenkins community is lively debating the future of the project (with the current release of Jenkins 2.0) and supportive of new ideas. This will insure that Jenkins remains relevant both for the main domain for which it was developed, as well as in new fields such as life and data sciences. I was impressed with Kohsuke’s dedication to the project, which send him around the country meeting with people like me to hear it from the ’trenches’. I hope it was useful for him, but I can certainly say that it has renewed my conviction that the powerful ideas that have been embedded into the Jenkins project and the support of the community will continue to spread Jenkins adaption and use in domains for which it was not originally envisioned. I know that this is already the case for many of my projects!