The Most Creative Stage in the Software Lifecycle?

Not too long ago I was working for a consulting company that prides itself in only accepting new development projects (as opposed to maintenance of existing systems). New development has an immediate appeal, no doubt. One would naturally expect that the blank slate that comes with the development of a brand new system would afford its builders the most freedom of expression and hence the most opportunity to be creative. I take it as a given that all talented individuals want a chance to be creative.

When I look back on my own career I see that I have spent approximately 50% working on brand new development and 50% on development within a system that was in place before I arrived. When I map these periods on to the periods where I felt most fulfilled as a developer, I notice that working on a completely new system was a poor predictor for happiness. It seems I was often being more creative doing, so called, maintenance. How can this be? Am I a ditch digger deep down inside?

Before I move forward, let me distinguish between two types of maintenance work. The first kind is on a system that it at the end of its life or is no longer considered strategic. Such a system must be maintained simply because the company still needs it in some capacity (the new system is not ready, the business is changing but not ready to abandon the income supported by the old or simply because that part of the business is still important but has no growth potential). The second kind is on a system that has existed for some time but is still considered strategic and is still evolving at a rapid pace. For the remainder of this essay, assume I am taking about the later system (if there are any readers out there who find joy at working on the first type of system, well, more power to you).

I maintain that there is often (and I mean very often) more opportunity to be creative evolving a strategic legacy system than there is on a clean slate system.

The first characteristic of clean slate development is great uncertainty. The customers are often not quite sure what they want. The development is as much an experiment as it is a development process except no one is going to call it a prototype because that is what was done six months ago by someone using Visual Basic + Excel before the team of 12 new expensive consultants were hired! It is always extremely frustrating to be told you are not building a throw away but on the other hand 60% of the system requirements are ill defined!

The second characteristic of clean slate development is great time pressure. You have a new development team that is often unproven and perhaps never worked together before. Everyone is anxious to prove their worth while the management team (possibly new as well) is anxious to show their bosses or clients results. High time pressure work is the least conducive to creativity!

The third characteristic of new development is new technology. After all, who wants to develop something new with something old? Often the catalyst for new development projects is a new technology that promises higher productivity at lower cost (C++ drove new development of systems originally written in C, Java did the same to both C and C++, .NET did the same to C++ and possibly Java and the cycle is sure to continue). However, new technology brings its own degree of uncertainty and creative impediments. It is tough to be creative from a problem solving point of view while at the same time discovering best practices for the new technology. Inevitably you get one or often both sides of the equation wrong!

Now contrast the so called legacy system development.

The working system is an operational specification. If you need to enhance something, you know what parts are supposed to change and what parts are supposed to remain the same. There may be uncertainty about the new but you are at least grounded in the experience of those who developed the old. Lessons have already been learned and those lessons are present in unambiguous code. Sure it may be spaghetti but even buggy spaghetti code is more deterministic than a customer who is only 30% sure what he wants!

The system (at least the part that exists) is presumably working to some degree. Perhaps it is too slow or perhaps it is too complex and sometimes it may crash but it basically works and supports the business. There are several highly creative acts that can occur in this context:

1. Figuring out how to make the system faster without breaking it.
2. Figuring out how to make the system more maintainable or flexible.
3. Reducing complexity while increasing robustness.

Very often these tasks have time pressure associated with them but the saving grace is that there is a working system to fall back on and it is unlikely the really creative problem solving that occurs during 1-3 is occurring under the glare of upper management.

Finally, you are released from the burden of new technology. This may sound very unappealing to you youngins out there but believe me, working on older technology can be very rewarding as long as it is not too old! I think the paradigmatic examples here are C++ for programming and SQL for databases . Both are rather ancient by computer standards but both are still evolving and have deep knowledge bases of best practices associated with them. Java probably has reached equivalent maturity at this time. The beautiful thing about being an expert is C++, Java and SQL is that you already paid your dues and can focus on the problem at hand and not the tool. Readers who have a hobby like carpentry or photography know what I am talking about. If you buy a router (I mean the kind for making fancy wood carvings) or a new fangled lens for your camera you know you are going to waste a significant time learning to use these before you actually enjoy using them.

Now, there are exceptions to everything, of course. I have had tremendous joy doing clean slate work but that was because the work was either on a very small team of very talented people or on a project that was understood by management to be more R than D. The real point of this article is to remind everyone that rewarding creative development often comes wrapped in quite unexpected packages.

Kindle Back in Stock

I'm cautiously optimistic again after Kindle shows up back in stock and Bezos letter to share holders is all about the Kindle and future innovation at Amazon. Fingers crossed.

Erlang is WTF?

As you know from a recent post, I am not into language bashing but I recently came across a witty one liner that sort of resonated with me: "C is fast, Ruby is beautiful and Erlang is WTF?" I might substitute a functional language (like Lisp or Haskel for Ruby) but the basic message is a valid one. Erlang as some cool ideas but they are packaged into a slow and rather ugly (syntactically) implementation.

Cargo Cults vs. Western Religion

Recently I came across an amusing phenomenon: Cargo Cults. A cargo cult is a religious movement that occurs when a primitive tribal culture comes in contact with a technological culture for the first time. These tribes come to believe that the technological artifacts of the encroaching culture really belong to them and they engage in rituals to coax their gods into giving them these material goods.

It is easy to laugh at these primitive fools but are modern western religions any more logical? Modern western religions (Judaism, Christianity, Islam) are grounded in the beliefs of humans who lived roughly 4000 years ago. The people of that time had significantly more technology than some of the tribes that form Cargo Cults but they were still very primitive by modern standards. One similarity is that western religions emerged either out of oppression or other hardship or as a result of an individual or individuals with delusions of grandeur. A primitive tribe must likewise feel disenfranchised when modern people show up on the scene with their wealth and gadgetry. The shamans (individuals with delusions of grandeur) of these cultures probably feel rather inadequate when their juju beans don't measure up to rifles and ocean freighters. But, why laugh at the very human reaction of the Cargo Cults and not at the founders of our "great" Western religions? It is really difficult for me to see the fundamental difference.

Intelligence = Prediction

The field of AI has always been a big draw for me. Understanding the processes that yield intelligent behavior has to be up near the top of the list of the BIG problems (up there with origin's of life and origins of the universe).

However, the underlying premises of mainstream AI research during the 80's (and even today) have always struck me as wrong headed. First Order Logic is useful if you want to create a software that behaves as if it has intelligence in a limited domain but is behavior really intelligence? Turing thought it was (Turing Test) but many philosophers disagreed and this lead to much heated but ultimately worthless debates of whether Chinese rooms can be intelligent, and the like.

Back in my graduate school days I wrote a thesis arguing that intelligence needs to be built on a foundation more akin to computer simulation than to logic based inference. My thinking was that simulation drives the ability to make predictions about what will happen in the world and predicting what will happen is a prerequisite to applying rules to act intelligently. Prediction precedes inference.

It was thus very refreshing to listen to Jeff Hawkins' talk at TED. He too argues for a definition of intelligence based on prediction. His research is focused more on new types of memory architectures than software but I think his work will be the foundation for the kinds of software solutions that will one day give us a HAL.

How do you know when you have mastered a new programming language

This is a minor continuation of my thoughts from The Law of the Excluded Middle does not apply to Programming Languages.

One sure sign that you have mastered a language is when you can create a fairly comprehensive list about what sucks about the language (while simultaneously appreciating why some of this suckiness is a necessary evil).

When you first meet a new language that floats your boat, there is a tendency is to fall in love. This happened to me not too long ago with Erlang. You think, "This language is great. Such and such is so hard to do in Language X and look how easy it is in Language Y.

Well, much like in the real world of human relationships you don't really know what love means until you get married! When you get married to a language (commit to developing a multi-year non-trivial system in it) then your love is surely tested. You learn about the languages warts and its tendency to leave its socks outside the hamper, squeeze the tooth paste from the top and leave the toilet seat in an inconvenient position!

If you still love the language with its warts and all, then you have some of the necessary (but not necessarily sufficient) hallmarks of a master. Either that, or you ave a really good therapist.

p.s. Erlang and I broke up but we are still friends! On a happier note, my long term mistress (Mathematica) and I are really making sparks fly! Hope C++ doesn't catch us.

The Second Law

Everything the educated layperson and puzzled student needs to know about one of the most important principles in physical science can be found at Prof. Frank L. Lambert's web site. Ten pages of low entropy enlightenment!

Staying Young

There is a large laundry list of characteristics that separate youth from old age but there is one characteristic in particular which has been poking around in my mind today. I think this characteristic is the most important because it applies to other entities, like companies, as much as it applies to people. And it is a characteristic that both companies and people can change about themselves.

I think one of the quintessential traits of youth is a tendency to focus on what can go right while old age focuses on what can go wrong. Sure, there are differences between individuals, some being optimists and some being pessimists. But, there is definitely a trend to become increasingly pessimistic with age. This is, of course, not unexpected. As we get older we have potentially much more to lose.

This same tendency applies to young versus old companies. Google versus Microsoft is an obvious comparison (as was Microsoft vs. IBM back in the day). Two young Turks with a few servers and an above average search engine have nothing to lose. Naturally they will eat drink and sleep what they can do to turn average into insanely great. In contrast, a ~20 yr old company that dominates the market for PC operating systems and application software has plenty to lose. A great deal of company resources must go into thoughts of protecting that turf.

Completely ignoring what can go wrong is called irresponsibility. No middle aged person or successful company can afford to focus 100% on the rosy scenarios. The question for both individuals and company leaders is "what do you want the dominant attitude to be"? The answer defines your age, your culture, your prospects for growth and ultimately the number of years you have left in this world (baring unforeseen tragedy).

I am presently working on a project where we are about to embark on some uncharted territory and it is pretty exciting. I am contractually prohibited on elaborating but I will say the following. The most frustrating aspect of working on this project is the overall dominating focus on what can go wrong. It is truly unbearable sometimes and even borders on the absurd . It should not surprise anyone that the work I am doing is for an older company.

So here is my 2 cents. You only have some many hours in a day and so many brain cells to occupy with thoughts. Be vigilant and make sure that at least 51% of your limited resources are focusing on success and improvement. You'll feel a lot younger.

The Law of the Excluded Middle does not apply to Programming Languages

While reading Herb Sutter's Blog I found a link in the comments to yet another rant about C++ and why this particular programmer does not use it any more. When are programmers going to wake up and stop this meaningless commentary about the merits of Programming Language A versus Programming Language B. When it comes to programming languages the Law of the Excluded Middle does not apply. The statements "C++ sucks" and "C++ is insanely great" are equally true. In fact you can say the same about Java, Perl, C#, etc.

It is not just that these statements can be true for one programmer and false for another. They can both be true to the same programmer on the same exact day. In fact, at this very moment I am compiling some C++ code and am of the opinion that C++ sucks and C++ is great. I think most programmers know what I am talking about. When a language does what you want or allows you to squeeze an extra uSec out of some code that needs said uSec squeezed it feels insanely great to have the privilege of working in that language. When a language gives you a completely cryptic page full of compiler errors before you squeeze out that uSec it really sucks.

This is really all that needs to be said to end all future language wars but like normal war, it will never stop. The average person just can't bare contradiction and this inevitably leads to conflict.

Do I contradict myself? Very well, then I contradict myself, I am large, I contain multitudes.

-- Walt Whitman.

Encyclopedia of Integer Sequences

I found this site via one of my connections on Linkedin. It is not the most intuitive web site in the world if you merely want to brows but it is rather unique and its search feature is probably useful to anyone doing Mathematics research.

An Open Letter to Jeff Bezos

Re: The Amazon Kindle

Dear Jeff,

I am writing to express my utter disappointment in Amazon's apparent lack of commitment to the Kindle. I am hoping my concerns are unfounded and that Amazon will do something in the near term to demonstrate that.

Let me explain. The day the Kindle was announced, I was ecstatic. I ordered it without any hesitation whatsoever. When asked by friends what I thought about the device, I sang its praises. You can find my review on your web site here. And here is an excerpt from an email I sent to my associates.

I read and buy a lot of books. I want the latest books and I don't have time to go to the library and when I do I always pay fines for returning them late. Anyway I like to write in my books so the library is out of the question. When Amazon offered their Amazon Prime service I jumped all over that because it would save me a ton in shipping cost and I can have the book I wanted in two days.

But the Kindle for me is 2 orders of magnitude better than Amazon Prime because:

1) I can have the book instantly.
2) The book is cheaper.
3) I am seriously running out of shelf space.
4) I can't carry more than 3 physical books with me on my 1.25 hour commute from beautiful Oyster Bay

So for me Kindle is not a device it is a SERVICE (yes I know I sound like Jeff B. but it is true).

Here is what this SERVICE gives me:

1) Instant access to all the books I buy hence forth.
2) The Wall St. Journal first thing in the morning (where I live your lucky if it gets there at 9:30 AM)
3) Instant access to a book I want to buy and a discount to boot.
4) Ability to search my books.
5) Ability to add notes.
6) All of this (including my notes) backed up on Amazon.
7) I can hold the Kindle in one hand and turn the pages while simultaneously drinking coffee with other hand while standing in the subway (THIS IS HUGE)!
8) I can read the WSJ without feeling like a complete idiot because I can't figure out how to turn the giant pages without whacking the girl sitting next to me on the train (THIS IS REALLY HUGE).

However, its been 5 months now and the Kindle platform seems to be stalled. Here is what I perceive as wrong:

1) The Kindle still shows itself sold out!
I realize this does not mean your have stopped producing and shipping Kindles. But to be in a perpetually sold out state for 5 months is not good. It may seem good to some but I am suspicious. The iPod is the most successful device in terms of sales in a long time and I don't believe Apple ever listed it as sold out on their web site. No matter how many iPods were purchased, Apple was committed to manufacturing more. There is no way you will have me believe that you are selling more Kindles than iPods, so why can't you keep any in stock?

2) There are a huge number of books I still have to buy as paper.
Why are you not being more aggressive in making deals with publishers like O'Reilly. I am told by my contacts there that your terms are too difficult. Open up your wallet and allow the publishers to make a buck and the Kindle will be the one of the most successful devices of all time (despite Job's pronouncement that no one reads anymore).

3) Why no software updates?
This device is far from complete! There is so many features you left out. Here is my list. I realize not all theses feature would be money makers for Amazon but some new features would show commitment to the platform.

a) Define my own short cut keys
b) Simple PDA functions like a calendar and todo list
c) A feature to turn on and off the wireless at specific times of the day to save battery. For example, I would turn mine on in the morning to receive the WSJ and then off.
d) Email client (besides gmail).
e) Better way to organize books such as user folders, short cut links, etc.
f) A customizable home screen (RSS feeds, weather, etc.)
g) A music player worth using!

I can go on and on.

Please tell me some of these things or better ones are in the works. Please tell me you did not get me all excited about the e-book revolution for nothing! Please stop being so secretive about sales figures. Please re-kindle my enthusiasm!

Sincerely,

Sal Mangano

Top 10 Foolish Beliefs

In honor of April 1st, traditionally April Fool's Day in the US and some other countries, here are my thoughts on the top 10 foolish beliefs of all time.

10. Perpetual motion is possible.

9. The Earth is Flat

8. My religion is correct, all others are wrong.

7. Dianetics (Scientology) is not a cult.

6. There is a Devil who lives in hell.

5. Astrology is a valid methodology for understanding your life.

4. Numerology has something relevant to say about our personal destiny.

3. Noah's Ark existed and had two of every animal on board.

2. Intelligent Design is a rational model of the origin of life.

1. God (as in an intelligent animate being) exists.

You may wonder why the Devil is only 6 while God is 1, besides the obvious numerological reason :-) . I justify this ranking based on the observation that there is much more anecdotal evidence for Satan than God!