The Most Creative Stage in the Software Lifecycle?

Not too long ago I was working for a consulting company that prides itself in only accepting new development projects (as opposed to maintenance of existing systems). New development has an immediate appeal, no doubt. One would naturally expect that the blank slate that comes with the development of a brand new system would afford its builders the most freedom of expression and hence the most opportunity to be creative. I take it as a given that all talented individuals want a chance to be creative.

When I look back on my own career I see that I have spent approximately 50% working on brand new development and 50% on development within a system that was in place before I arrived. When I map these periods on to the periods where I felt most fulfilled as a developer, I notice that working on a completely new system was a poor predictor for happiness. It seems I was often being more creative doing, so called, maintenance. How can this be? Am I a ditch digger deep down inside?

Before I move forward, let me distinguish between two types of maintenance work. The first kind is on a system that it at the end of its life or is no longer considered strategic. Such a system must be maintained simply because the company still needs it in some capacity (the new system is not ready, the business is changing but not ready to abandon the income supported by the old or simply because that part of the business is still important but has no growth potential). The second kind is on a system that has existed for some time but is still considered strategic and is still evolving at a rapid pace. For the remainder of this essay, assume I am taking about the later system (if there are any readers out there who find joy at working on the first type of system, well, more power to you).

I maintain that there is often (and I mean very often) more opportunity to be creative evolving a strategic legacy system than there is on a clean slate system.

The first characteristic of clean slate development is great uncertainty. The customers are often not quite sure what they want. The development is as much an experiment as it is a development process except no one is going to call it a prototype because that is what was done six months ago by someone using Visual Basic + Excel before the team of 12 new expensive consultants were hired! It is always extremely frustrating to be told you are not building a throw away but on the other hand 60% of the system requirements are ill defined!

The second characteristic of clean slate development is great time pressure. You have a new development team that is often unproven and perhaps never worked together before. Everyone is anxious to prove their worth while the management team (possibly new as well) is anxious to show their bosses or clients results. High time pressure work is the least conducive to creativity!

The third characteristic of new development is new technology. After all, who wants to develop something new with something old? Often the catalyst for new development projects is a new technology that promises higher productivity at lower cost (C++ drove new development of systems originally written in C, Java did the same to both C and C++, .NET did the same to C++ and possibly Java and the cycle is sure to continue). However, new technology brings its own degree of uncertainty and creative impediments. It is tough to be creative from a problem solving point of view while at the same time discovering best practices for the new technology. Inevitably you get one or often both sides of the equation wrong!

Now contrast the so called legacy system development.

The working system is an operational specification. If you need to enhance something, you know what parts are supposed to change and what parts are supposed to remain the same. There may be uncertainty about the new but you are at least grounded in the experience of those who developed the old. Lessons have already been learned and those lessons are present in unambiguous code. Sure it may be spaghetti but even buggy spaghetti code is more deterministic than a customer who is only 30% sure what he wants!

The system (at least the part that exists) is presumably working to some degree. Perhaps it is too slow or perhaps it is too complex and sometimes it may crash but it basically works and supports the business. There are several highly creative acts that can occur in this context:

1. Figuring out how to make the system faster without breaking it.
2. Figuring out how to make the system more maintainable or flexible.
3. Reducing complexity while increasing robustness.

Very often these tasks have time pressure associated with them but the saving grace is that there is a working system to fall back on and it is unlikely the really creative problem solving that occurs during 1-3 is occurring under the glare of upper management.

Finally, you are released from the burden of new technology. This may sound very unappealing to you youngins out there but believe me, working on older technology can be very rewarding as long as it is not too old! I think the paradigmatic examples here are C++ for programming and SQL for databases . Both are rather ancient by computer standards but both are still evolving and have deep knowledge bases of best practices associated with them. Java probably has reached equivalent maturity at this time. The beautiful thing about being an expert is C++, Java and SQL is that you already paid your dues and can focus on the problem at hand and not the tool. Readers who have a hobby like carpentry or photography know what I am talking about. If you buy a router (I mean the kind for making fancy wood carvings) or a new fangled lens for your camera you know you are going to waste a significant time learning to use these before you actually enjoy using them.

Now, there are exceptions to everything, of course. I have had tremendous joy doing clean slate work but that was because the work was either on a very small team of very talented people or on a project that was understood by management to be more R than D. The real point of this article is to remind everyone that rewarding creative development often comes wrapped in quite unexpected packages.