Saturday, April 24, 2010

Lets go build some obsolete tools.... and prevent being blamed.

One of the first stages in the development of a new tool (software or hardware) is a functional specification. The functional specification matures in discussions between the developers (department of R&D) and the customer representatives (often the departments of marketing and sales).

Of course a functional specification is useful: it is very hard to develop something new without an idea of how the new tool will be used and what it will be compared with. However, defining a functional specification can also be taken too far. In some organizations, the functional specifications are spelled out in the tiniest details. At the end of a long formal procedure, the book of specifications is signed like a contract between marketing and development. The development can only be started when the list of signatures is complete and will be performed in splendid isolation from the world of potential users. Why do organizations do specifications this way? It is often an attempt to separate the responsibilities of the departments, so that if anything fails the appropriate party can be blamed. If the final product does not meet the functional specifications, this can be blamed on the developers. And if the product does not succeed even though it meets all the functional specifications, this will be blamed on marketing.

I have a serious problem with this approach: using this procedure, how can one ensure that the product will be useful? After two years without interaction, development may produce exactly what marketing asked for (making all deadlines and within budget), but the market has changed and does no longer need the designed product. Or technology has changed, and better specifications would have been within reach and are offered by the competition. Or maybe marketing truly made a mistake, and asked for something the world is not waiting for. In these cases, clearly the development department can not be blamed! But if you develop an obsolete product this way, where does that leave the organization as a whole? And if this is not the best solution for the organization as a whole, will it be good for the development department? Even though everyone did exactly what was expected, people may be laid off because development costs can not be recovered.

The solution is, as often, to keep a middle road. Using e.g. Agile or LEAN development methods,  developers can stay in constant communication with marketing. Iterative  and modular design procedures can be used to verify that the new tool does what it should, without relying on the capacity of people to describe specifications in words beforehand. And because the communication with the market is not lost during the development process, the tool will have a significantly higher chance of actually being useful at the moment of introduction.

Image (by QuiteLucid on Flickr): "a camel is a racing horse designed by a committee".

Monday, April 19, 2010

Abstract or concrete? It all depends on your point of view.

Our team is working with a number of task forces in bioinformatics. Each of those task forces was started to collaborate on the development of a platform for their sub-field: a set of software tools that work together to solve the problems that everyone in the field needs to solve. Developing the platform does not require any new bioinformatics developments: the purpose is to put existing tools together.

The advantage of having these platforms available is obvious:
  • to a biologist the advantage consist of having all the de facto standard tools available under the press of a button.
  • to a specialist bioinformatics researcher working on a new tool the advantage is that he does not have to deal with the intricacies of all the other tools, and is able to plug his new tool into the platform using well described protocols.
To get to the development of such a platform there is a bootstrapping problem. The situation is like a table with biologists sitting on one side, bioinformaticians at the other side. Above the table, a thick (volcanic?) fog. The layout of the platform is drawn in diagrams on the table: all the tools making up the common work flow, with all their relations. On the side of the bioinformaticians, the diagram shows the concrete tools. Through the fog, they can vaguely see the workflow on the other side of the table. For the biologists, the situation looks completely different: they have a clear view on the concrete workflow they need, but the tools are vague entities that are only visible through the thick fog.

Without good support from a project leader that can listen to people on both sides of the table, the bioinformaticians will try to solve the very concrete problems they encounter on their very concrete individual tools. A little optimization here, a better data storage facility there. None of this is visible for the biologists.

This is why we put project leaders from our engineering team into each of the task forces. They will direct the focus of the bioinformaticians towards more visible changes. Work on common data formats. Work on (common) user interfaces.

Getting things to work together will bootstrap the true collaborative advantages. It will blow away the fog. Suddenly the biologists will be able to see what is going on. They will be able to provide directed feedback. And the bioinformaticians will be able to see the workflow even from their side, and build upon it.

Image credit: Three views of three tables, by EJP Photo on Flickr.