trystan.org

The are many blog musings around this topic. However, most seem to reiterate the software cycle merits of agile processes without attention to how organizational structure and the ingrained psychology of executives can derail functional implementations of either process. Waterfall methodologies are easy for people with business backgrounds, and those otherwise steeped in corporate culture, to understand. A waterfall process is in fact a reflection of a corporate hierarchical structure in form and function. In the software world, this can actually work when the people at the top of the waterfall have some clue regarding the architecture or component level implementation of their own system. However, on many occasions, I’ve witnessed a very top down/waterfall approach to software process imposed by upper management where the people driving development are oblivious to the overall workings of the system. In my experience, this is one scenario that leads to an anti-pattern of continuous reinvention and re-planning by technologically naive executives and upper management. The flow is something like the following dialog, “We [business leadership] have problem X with the technology. Let’s take a couple weeks and talk to the dev leads and hammer out a hypothetical solution.” A couple weeks later the business leadership, thinking they actually grasp the technology after a few meetings, come up with plan 9 from outer space. Often, the plans make sense when one considers their level of understanding. However, it’s doomed from the beginning because there’s not enough depth to build a realistic schedule around. Eager, fearful or equally naive persons in middle management sign off on the plan and ‘insulate’ individual contributors from the insanity. Depending on the size of the system and organization, the infeasibility of plan 9 may take weeks or months to be fully realized. Upper management becomes frustrated with the lack of progress, assume it’s due to people being at slightly wrong places in the hierarchy, rejigger the org structure a bit and repeat; never addressing the underlying issue of unhealthy software processes.

In the type of organization where business is very separate from development and upper management does not want to know how the system works at a functional level of detail, agile practices are of critical importance. The situation above can not occur in an environment where top down process is replaced with grassroots process. A fundamental notion of agile methods, and probably one of the most overlooked, is decisions impacting an individual contributor’s area of responsibility are made by that contributor. In this model, management plays a role more akin to ‘conflict mitigation’ than process driver. Their primary responsibility becomes integrating potentially conflicting perspectives coming from below rather than attempting to subdivide a misguided perspective from above. The main advantage here is that individual contributors have the appropriate level of visibility to a problem sub-domain which is virtually impossible for management to collectively assimilate. Unfortunately, the corporate psychology of hierarchical organization dreadfully inhibits the evolution of this type of process.

I have heard of cases where a company seemed to implement the type of organization described above successfully. Some use Toyota’s ‘lean’ methodology as an example. However, I find that an analogy between manufacturing carries limited validity with software development as I’m convinced that development is essentially research. There is no physical line where something is guaranteed to come out at the end. Software development is capable of devouring an infinite amount of resources with no predictable outcome. One thing I’m certain of at this point is an effective software organization, at least one dealing with new problems, can not function in a purely top-down fashion. A future that seems likely to me is one where corporations do less and less software development in house and rely on organization that do only software. These software-only organizations have more freedom to abandon a traditional hierarchical structure in favor of a new organizational structure favoring more efficient development. Such a company may differ from traditional contractors in that they may maintain the software indefinitely for the client.

As a somewhat radical leftist I often argue with associates, typically in a friendly context, regarding the semantics of “freedom” within the domain of archetypal political/economic systems such as capitalism versus socialism. An argument often delivered by proponents of capitalism is that a person should have the freedom to own resources and modify them into consumables for a net profit if that is what satisfies them in life. I argue to the contrary on the grounds that this process is inherently exploitative and therefore eventually oppressive with respect to others. Ultimately, it is detrimental to the exploiters themselves. The point expressed here is not purely my own but a product of my own ideas and others via casual conversation.

To begin I’ll borrow a model put forth by a friend that simply states, “one key failure of the current incarnation of the market economy is the failure to integrate all the factors contributing to the ‘cost’ of a product intended for consumption.” For instance, a simple supply vs demand model does not include the cost of resources from private property such as timber. The cost of extraction my be included but perhaps not the cost of the water used to replant the trees. This in of itself is not a criticism of capitalism per say but rather appeals to one to acknowledge that we have not developed models sophisticated enough to properly price consumables. As a result, the system is entropic or “leaky.”

I propose that if a realistic value was applied to all variables within the consumption chain the system would prove to be unsustainable given the current rate of consumption. That is, the current incarnation of capitalism depends on unchecked exploitation of resources or the inequitable exchange of resources at some point in the chain. A system that included the cost for items not currently valued or undervalued would cease to yield a net profit for the capitalist and then capitalism crumbles.

On this train of thought there are two destinations. First, capitalism can continue along the current course and over the long term, if you buy my argument, the system will collapse either exhausting a key undervalued resource(s), or decay into something resembling an oligarchy where only an elite class have access to a comfortable lifestyle by today’s standards. The lack of scientific valuation of key resources has consequences visible in the speculative oil price run-up of late. Additionally, the largest corporations already represent something of an oligarchy in terms of their influence over the lives of large populations.

The alternative is striving for a more accurate valuation of resources and scaling back lifestyles accordingly. If a net profit is still possible in areas of this new model then capitalism may still survive. This view is held by many who favor further privatization, believing that approach will lend itself to more accurate valuation. However, at least by appealing to my own intuition and knowledge of closed systems, I fail to see how that could be possible. Using the first law of thermodynamics as a rough analogy, resources(energy) can change hands(form) indefinitely but there’s no net increase(profit) in the system. This, along with some humanist reasoning, is why I favor systems based on cooperation as opposed to exploitation.

“Because the universe is expanding, it takes me longer each morning to find my keys.” – Woody Allen