The Most Creative Stage in the Software Lifecycle?

Not too long ago I was working for a consulting company that prides itself in only accepting new development projects (as opposed to maintenance of existing systems). New development has an immediate appeal, no doubt. One would naturally expect that the blank slate that comes with the development of a brand new system would afford its builders the most freedom of expression and hence the most opportunity to be creative. I take it as a given that all talented individuals want a chance to be creative.

When I look back on my own career I see that I have spent approximately 50% working on brand new development and 50% on development within a system that was in place before I arrived. When I map these periods on to the periods where I felt most fulfilled as a developer, I notice that working on a completely new system was a poor predictor for happiness. It seems I was often being more creative doing, so called, maintenance. How can this be? Am I a ditch digger deep down inside?

Before I move forward, let me distinguish between two types of maintenance work. The first kind is on a system that it at the end of its life or is no longer considered strategic. Such a system must be maintained simply because the company still needs it in some capacity (the new system is not ready, the business is changing but not ready to abandon the income supported by the old or simply because that part of the business is still important but has no growth potential). The second kind is on a system that has existed for some time but is still considered strategic and is still evolving at a rapid pace. For the remainder of this essay, assume I am taking about the later system (if there are any readers out there who find joy at working on the first type of system, well, more power to you).

I maintain that there is often (and I mean very often) more opportunity to be creative evolving a strategic legacy system than there is on a clean slate system.

The first characteristic of clean slate development is great uncertainty. The customers are often not quite sure what they want. The development is as much an experiment as it is a development process except no one is going to call it a prototype because that is what was done six months ago by someone using Visual Basic + Excel before the team of 12 new expensive consultants were hired! It is always extremely frustrating to be told you are not building a throw away but on the other hand 60% of the system requirements are ill defined!

The second characteristic of clean slate development is great time pressure. You have a new development team that is often unproven and perhaps never worked together before. Everyone is anxious to prove their worth while the management team (possibly new as well) is anxious to show their bosses or clients results. High time pressure work is the least conducive to creativity!

The third characteristic of new development is new technology. After all, who wants to develop something new with something old? Often the catalyst for new development projects is a new technology that promises higher productivity at lower cost (C++ drove new development of systems originally written in C, Java did the same to both C and C++, .NET did the same to C++ and possibly Java and the cycle is sure to continue). However, new technology brings its own degree of uncertainty and creative impediments. It is tough to be creative from a problem solving point of view while at the same time discovering best practices for the new technology. Inevitably you get one or often both sides of the equation wrong!

Now contrast the so called legacy system development.

The working system is an operational specification. If you need to enhance something, you know what parts are supposed to change and what parts are supposed to remain the same. There may be uncertainty about the new but you are at least grounded in the experience of those who developed the old. Lessons have already been learned and those lessons are present in unambiguous code. Sure it may be spaghetti but even buggy spaghetti code is more deterministic than a customer who is only 30% sure what he wants!

The system (at least the part that exists) is presumably working to some degree. Perhaps it is too slow or perhaps it is too complex and sometimes it may crash but it basically works and supports the business. There are several highly creative acts that can occur in this context:

1. Figuring out how to make the system faster without breaking it.
2. Figuring out how to make the system more maintainable or flexible.
3. Reducing complexity while increasing robustness.

Very often these tasks have time pressure associated with them but the saving grace is that there is a working system to fall back on and it is unlikely the really creative problem solving that occurs during 1-3 is occurring under the glare of upper management.

Finally, you are released from the burden of new technology. This may sound very unappealing to you youngins out there but believe me, working on older technology can be very rewarding as long as it is not too old! I think the paradigmatic examples here are C++ for programming and SQL for databases . Both are rather ancient by computer standards but both are still evolving and have deep knowledge bases of best practices associated with them. Java probably has reached equivalent maturity at this time. The beautiful thing about being an expert is C++, Java and SQL is that you already paid your dues and can focus on the problem at hand and not the tool. Readers who have a hobby like carpentry or photography know what I am talking about. If you buy a router (I mean the kind for making fancy wood carvings) or a new fangled lens for your camera you know you are going to waste a significant time learning to use these before you actually enjoy using them.

Now, there are exceptions to everything, of course. I have had tremendous joy doing clean slate work but that was because the work was either on a very small team of very talented people or on a project that was understood by management to be more R than D. The real point of this article is to remind everyone that rewarding creative development often comes wrapped in quite unexpected packages.

Kindle Back in Stock

I'm cautiously optimistic again after Kindle shows up back in stock and Bezos letter to share holders is all about the Kindle and future innovation at Amazon. Fingers crossed.

Erlang is WTF?

As you know from a recent post, I am not into language bashing but I recently came across a witty one liner that sort of resonated with me: "C is fast, Ruby is beautiful and Erlang is WTF?" I might substitute a functional language (like Lisp or Haskel for Ruby) but the basic message is a valid one. Erlang as some cool ideas but they are packaged into a slow and rather ugly (syntactically) implementation.

Cargo Cults vs. Western Religion

Recently I came across an amusing phenomenon: Cargo Cults. A cargo cult is a religious movement that occurs when a primitive tribal culture comes in contact with a technological culture for the first time. These tribes come to believe that the technological artifacts of the encroaching culture really belong to them and they engage in rituals to coax their gods into giving them these material goods.

It is easy to laugh at these primitive fools but are modern western religions any more logical? Modern western religions (Judaism, Christianity, Islam) are grounded in the beliefs of humans who lived roughly 4000 years ago. The people of that time had significantly more technology than some of the tribes that form Cargo Cults but they were still very primitive by modern standards. One similarity is that western religions emerged either out of oppression or other hardship or as a result of an individual or individuals with delusions of grandeur. A primitive tribe must likewise feel disenfranchised when modern people show up on the scene with their wealth and gadgetry. The shamans (individuals with delusions of grandeur) of these cultures probably feel rather inadequate when their juju beans don't measure up to rifles and ocean freighters. But, why laugh at the very human reaction of the Cargo Cults and not at the founders of our "great" Western religions? It is really difficult for me to see the fundamental difference.

Intelligence = Prediction

The field of AI has always been a big draw for me. Understanding the processes that yield intelligent behavior has to be up near the top of the list of the BIG problems (up there with origin's of life and origins of the universe).

However, the underlying premises of mainstream AI research during the 80's (and even today) have always struck me as wrong headed. First Order Logic is useful if you want to create a software that behaves as if it has intelligence in a limited domain but is behavior really intelligence? Turing thought it was (Turing Test) but many philosophers disagreed and this lead to much heated but ultimately worthless debates of whether Chinese rooms can be intelligent, and the like.

Back in my graduate school days I wrote a thesis arguing that intelligence needs to be built on a foundation more akin to computer simulation than to logic based inference. My thinking was that simulation drives the ability to make predictions about what will happen in the world and predicting what will happen is a prerequisite to applying rules to act intelligently. Prediction precedes inference.

It was thus very refreshing to listen to Jeff Hawkins' talk at TED. He too argues for a definition of intelligence based on prediction. His research is focused more on new types of memory architectures than software but I think his work will be the foundation for the kinds of software solutions that will one day give us a HAL.

How do you know when you have mastered a new programming language

This is a minor continuation of my thoughts from The Law of the Excluded Middle does not apply to Programming Languages.

One sure sign that you have mastered a language is when you can create a fairly comprehensive list about what sucks about the language (while simultaneously appreciating why some of this suckiness is a necessary evil).

When you first meet a new language that floats your boat, there is a tendency is to fall in love. This happened to me not too long ago with Erlang. You think, "This language is great. Such and such is so hard to do in Language X and look how easy it is in Language Y.

Well, much like in the real world of human relationships you don't really know what love means until you get married! When you get married to a language (commit to developing a multi-year non-trivial system in it) then your love is surely tested. You learn about the languages warts and its tendency to leave its socks outside the hamper, squeeze the tooth paste from the top and leave the toilet seat in an inconvenient position!

If you still love the language with its warts and all, then you have some of the necessary (but not necessarily sufficient) hallmarks of a master. Either that, or you ave a really good therapist.

p.s. Erlang and I broke up but we are still friends! On a happier note, my long term mistress (Mathematica) and I are really making sparks fly! Hope C++ doesn't catch us.

The Second Law

Everything the educated layperson and puzzled student needs to know about one of the most important principles in physical science can be found at Prof. Frank L. Lambert's web site. Ten pages of low entropy enlightenment!