Programming like it is still Middle Ages, Part 2 - Software Factory Pipe Dreams

Keeping in mind the discussion of modes of production (in part 1) we're going to take a closer look at the poster child of post-industrialism: modern software development.

It is hardly a secret that the shiny facade of high technology hides some truly ugly problems: first of all, the success rate of software projects is abysmally low: a relatively recent study found that as many as 68% of all software/IT projects fail.  The quality of software (although much harder to quantify) is universally poor, especially where security is concerned.  The incredible growth of CPU performance and amounts of memory and data storage are not matched by visible improvements in software responsiveness and usability. To top it off, a significant part of software developers cannot even be described as competent programmers despite being able to negotiate rather impressive wages, indicating both the increasingly dire shortage of software developers and fundamental brokenness of the hiring and management practices.

To get some insight into the nature of the problem  (the necessary first step in being able to come up with a serviceable solution to it) we need to take a closer look at how software is produced.  At a risk of sounding like Captain Obvious I would like to state that each mode of production has its own best methods of organizing it.  These methods are evolved products of selective pressures (those which worked were imitated, these which didn't failed and disappeared).  Being confused about which mode of production is involved leads to the failed attempts to organize and manage production is ways which will be consistently deleterious or even catastrophic.

The common wisdom is that high-tech in general and software development in particular are a shiny new production mode: the wonderfully vague "post-industrial".  This unfortunately yields no practical insight into what it is: what is the relation of labor, capital, and inputs in this mode of production?  Let's try to characterize these based on what we see in reality.

We need to keep in mind that production cycle in software is rather long (many months or years), and this somewhat complicates teasing out which factors are capital and which are inputs.  The depreciation time of some nominally capital factors (such as computers) is close enough to the length of the production cycle to consider them consumable inputs instead.

Labor: Software development is very labor-intensive, and budgets of high-tech companies are dominated by labor costs.  A modern software project could easily be from 100K to 10M lines of code (however flawed this metric is, it at least offers some way of coming up with order-of-magnitude estimates).  A recent estimate of an average software developer's productivity  comes up to something like 325 to 750 lines of code per month (Jones, Capers and Bonsignour, Olivier. The Economics of Software Quality.).  For a small 2-year 100K LOC project it implies staff of 6-12 developers, for a 10-year 10M LOC project the estimate would be around 300 developers (in reality the headcount will be much more because productivity rapidly falls with increasing complexity, the staff of 1500-3000 engineers for a project on this scale would be closer to reality).

Besides hours spent, labor in high-tech contributes personal knowledge and experience, which are costly to acquire.  I.e. the "talent" contributes capital in addition to labor, which is tacitly recognized by the structure of compensation in the high-tech industry which commonly gives equity in the business (in form of stock options) to the engineers as a part of the package in order to create incentive for the "human capital" to stay longer with the company.

Capital: The capital goods used in production of software are mostly in the form of the office buildings, institutional knowledge, and acquired intellectual property (IP).  Offices used in software development are not any different from pretty much any other office space, which is practically a commodity; most software businesses rent.  Of course, a start-up company needs financial capital in order to pay labor and other costs before it becomes profitable, but mature software companies do not need outside financial capital to sustain their production.

Acquired IP is not as important as it is appears to be because the code itself is not of much practical use without people who understand it.  Gaining understanding of somebody else's code to the point of being able to significantly modify and successfully maintain it long-term requires efforts comparable to writing the code from scratch (especially if code quality isn't great and supporting documentation is lacking). As it happens, IP critical for the actual software development is mostly in public domain and/or free. (Why is this so is outside of scope of this post).

The only really important capital good necessary for software development is knowledge created in-house, usually as a byproduct of creating the software.  Unfortunately, most companies are not good in promoting accumulation and preservation of this knowledge, and often create perverse incentives which inhibit knowledge transcription and sharing (again, this is a separate discussion) - the situation eerily resembling the reluctance of medieval artisans to share their knowledge.

Inputs: the inputs in software development (besides trifles such as coffee and electricity to run computers in the office) are quickly depreciating computer hardware, licensed IP, on-line product service fees, and rental office space.  These are relatively minor lines in the budgets.

The final observation about production of software is that while software costs next to nothing to replicate, producing it is always custom work.  Where production is concerned, all software is bespoke - no two items are the same.  In fact, there's not much standardization in software engineering: there are some standards for programming languages and network protocols, but the practical implementations are never 100% compliant, and a common practice among the vendors is to deliberately break compatibility by creatively "extending" the standards for the purpose of customer lock-in.

Just like there is notable paucity of software object standardization, there is also notable lack of standards for professional qualification.  This leads to the strange spectacle of academia churning out massive numbers of "computer science" graduates who lack the skills and knowledge to program in the real world.  This knowledge is then acquired through on-the-job training, which ranges from autodidact learning to apprenticeship.

All of the above strongly matches the artisanal mode of production: the most advanced and complex artifacts of human civilization are, in fact, produced in the way not much different from the artisans of old firing clay pots and smiting horseshoes.  In no way it is "industrial", leave alone "post-industrial".  We program like it's still Middle Ages.

This, however, does not stop the business people to try to impose management methods developed for industrial production (after all, this is what they are taught in the business schools) onto software development, with little success.  No amount of wishful thinking could make an occupation requiring advanced abstract reasoning into blue-collar work. The "software factory" is still a pipe dream, by now becoming firmly associated with outsourced production of junk software on the cheap. It may actually be better to re-create the social structures which evolved during Middle Ages to support the artisanal production.  Yes, guilds (a historical tidbit: guilds created modern academia by establishing the early secular universities).

Another feature of artisanal production is its unpredictability: the lack of standards and high variance in professional quality (at least guilds provided some guarantees about performance of their members) makes planning of software development an exercise in futility and outright fraud: quite a lot of software engineering projects are delivered "on-time" by means of cutting corners, changing the definition of what is going to be delivered, and compromising on quality and future maintainability of the delivered code.

The problem with the artisanal mode of production is that it does not scale well, and no management fads are going to change this basic fact.  We cannot deal with the crisis in software production without changing the nature of software development by figuring out why our attempts to mechanize software production and to standardize software objects were such abject failures (and not for the lack of trying by very smart people) and finding ways to overcome this obstacle.  This will be the main theme of many future posts in this blog.