Frederick Phillips Brooks' seminal work, "The Mythical Man-Month," explores the challenges of managing software projects and presents the idea that adding more manpower to a late software project only makes it later due to the complexities of communication and coordination. The work has transcended time and now requires a fresh edition as we delve into the era of Mysterious Machine Month. While Brooks, a Turing Award winner, isn't here to guide us through this transition, the landscape has evolved significantly. Machines differ fundamentally from humans in their approach to work; they are devoid of procrastination and exhibit a consistently high level of productivity, making them indispensable in the world of technology.
The future of the tech industry hinges on embracing the Machine Month paradigm. However, this transition is not without its challenges and myths that need to be dispelled. Machine complications are uncharted territory, and if not managed right can be a lot more convoluted than complications created by their human counterparts. Above that machines still rely on their human partners, which can transmit biases adding to the existing complexity. Investment tech is an important case study to illustrate the transition of myth of man month to mystery of the Machine Month. Machines if not managed well can add to the already significant volatility in global markets.
The Mythical Man
Brooks, the renowned American computer architect and software engineer, etched his name in the annals of computer science history through his work on IBM's System/360 family of computers and OS/360 software support. Unfortunately, he passed away last November, but his contributions, including his 1999 Turing Award, continue to inspire. His induction into the National Academy of Engineering was a testament to his significant impact on computer system design.
Obsolescence
Brooks astutely noted, that Large systems were destined to be plagued by problems highlighting the vexing reality of obsolescence in the world of technology. The relentless march of technology ensures that even as we labor on a project, it inches closer to obsolescence. In the new age, obsolescence extends beyond programming into the realm of investment instruments and methodologies. The Machine Age has brought us to the precipice of a failure cliff where redundancy, inefficiency, and concentration are destined to fall by the wayside. Machines demand results, and Nobel Prize-winning theories that lack practical application are no longer tenable. Machines must excel at building entropy-decreasing processes rooted in Modern Science, rather than indulging in psychological storytelling of anomalies.
Conceptual Integrity
Today's era affords us the luxury of focusing on architectural integrity, planning, and system design. Conceptual integrity, as Brooks rightly pointed out, leads to simplicity and straightforwardness. The essence of programming lies in effective representation, and the ability to iterate rapidly, acknowledging failures candidly. For machines, the granularity of code is immaterial; they excel at comprehending both the micro and macro aspects of systems. Even the concept of "throw one away" is different for machines, because for it everything is a large language model. Hence unlike Man, the machines are more adept the evolving nature of design, as bugs resurface and new defects emerge.
Intelligent Investment Tech
Models and statistics predate the era of machines. Any large investment technology hence is a good case study to understand the mystery of machine months. In this age of hyper-personalization, every investor deserves tailored solutions, which needs automation. Machines must process, replicate, maintain, and compute at varying periodicities while adapting to changing preferences. Such Machines are accessible to every one today. Model building is a matter of few queries. Anyone with a little knowledge can build quantiative models, simulate them and execute. In such a world, it is easy to lose the definition of intelligence. But then just because everyone has a computer, does not naturally translate to an intelligence society. Making a quant out of everyone does not translate into sustainable alpha. This is how machines months become mysterious and supersede the complexity and noise created by the man months. Building large investing machines is a miracle of the modern world because it will have to operate among millions of human driven machines, creating a noise of their own, heralding a new chaotic world, where it will become harder for active managers to perform. As the high-frequency investing world experiences an exponential surge in noise, it tilts the balance even more towards slower passive investing.
Entropy, Computation, and Dynamic Design
Drawing parallels to the realm of investment tech as large-scale systems programming, computational efficiency becomes a critical concern. Brooks' insights into entropy-decreasing processes underscore the inherent stability in system building. However, maintaining this stability necessitates efficient program maintenance and constant adaptation. In the age of AI, computational efficiency becomes intertwined with risk management, emphasizing the need to balance environmental considerations with processing demands. Computation must be seen as an information process, requiring entropy reduction for efficiency. Brooks' principles of generalization and efficient design hold true in this context, allowing large-scale systems to function optimally.
Entropy, Computation, and Adaptive Design
Building on the case of investment tech as large-scale systems programming, it is not hard to realize the need for computationally efficient systems, both for processing information and delivering these solutions to global investors. Brooks highlighted in his seminal work how systems program building is an entropy-decreasing process, hence inherently metastable. Program maintenance, on the other hand, is an entropy-increasing process, and even its most skillful execution merely postpones the subsidence of the system into unfixable obsolescence. The ever-increasing influx of information makes it crucial to prioritize computational efficiency. Notably, Brooks does not explicitly mention computation in his body of work. However, informational entropy and computational efficiency do not exist in mutually exclusive worlds. Any large-scale technology of today and tomorrow must master both entropy and computation to succeed.
Considering the case study of investment tech, it becomes evident that creating a climate-friendly product for an investor while doubling the carbon footprint is untenable. Reducing both computation and entropy necessitates dynamic design, generalization, and simplicity. As Brooks wisely noted, "To craft a universally applicable programming product, one must construct it with a mindset of generality." In simpler terms, we can invoke Brooks's Law: "Adding manpower to a late software project only serves to further delay its completion." Integral to this efficient design is meticulous resource budgeting, encompassing memory allocation, input-output device management, and judicious utilization of computational time.
In the ever-evolving landscape of technology, the moment one solidifies a design, it becomes conceptually obsolete. Thus, arises the need for a meta-design approach that perpetually explores alternative pathways and innovative solutions. In this dynamic and ever-shifting milieu, adaptability reigns supreme.
The Unpredictable Nature of Progress
Blaise Pascal once remarked, "'Things are always at their best in the beginning,'" and C. S. Lewis articulated this phenomenon more astutely as the key to history. We witness a tremendous outpouring of energy, the construction of civilizations, and the creation of exceptional institutions. However, inevitably, something goes awry. A fatal flaw emerges, propelling selfish and cruel individuals to positions of power, resulting in a descent back into misery and ruin. It's as if the grand machine of progress, having initially fired up with promise, suddenly conks out. It starts with vigor, covering a few yards, only to break down.
Machines, despite their technological prowess, are not immune to the curse associated with largeness. Nonetheless, they possess a distinct advantage when it comes to circumventing the Mythical Man-Month curse. Their lack of emotions, ability to operate without biases, reduced communication challenges, and comfort with abstract thinking enable them to embrace complexity and speed with unmatched efficiency. Unlike humans, machines do not demand vacations, seek likability, require entertainment, or necessitate translation services. In the world of machines, there is no Tower of Babel-like collapse.
However, even as machines outperform humans in numerous aspects, they remain, in many ways, extensions of their human creators. This inherent connection and the combination of AI makes them inherently more unpredictable and mysterious. Ultimately, technology that excels in generalization, simplification, entropy reduction, computational efficiency, and adaptability will triumph over the paradox that afflicts large-scale tech
Notes
“Systems program building is an entropy-decreasing process, hence inherently metastable. Program maintenance is an entropy- increasing process, and even its most skillful execution only delays the subsidence of the system into unfixable obsolescence.”
“Obsolescence is a reality not just for programming but for Passive instruments and methodologies. We have seen it happening with Nobel Prize in economics again and again.(*notes) The only way to remain relevant is by building entropy decreasing processes.”
“The last woe, and sometimes the last straw, is that the product over which one has labored so long appears to be obsolete upon (or before) completion…”
“Already the displacement of one's thought-child is not only conceived, but scheduled. Of course the technological base on which one builds is always advancing“
“To become a generally usable programming product, a program must be written in a generalized fashion.”
“Oversimplifying outrageously, we state Brooks's Law: Adding manpower to a late software project makes it later.”
“Because now the world can actually afford a few good minds doing design and construction.”
“The dilemma is a cruel one.”
“These studies revealed large individual differences between high and low performers, often by an order of magnitude.”
“In short the $20,000/year programmer may well be 10 times as productive as the $10,000/year one.”
“To make that job manageable, a sharp distinction must be made between architecture and implementation, and the system architect must confine himself scrupulously to architecture.”
“conceptual integrity is the most important consideration in system design.”
“It is not enough to learn the elements and rules of combination; one must also learn the idiomatic usage, a whole lore of how the elements are combined in practice. Simplicity and straightfor- wardness proceed from conceptual integrity.”
“Representation is the essence of programming.”
“It is common sense to take a method and try it. If it fails, admit it frankly and try another. But above all, try something.”
“Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowcharts; they'll be obvious.”
“The discard and redesign may be done in one lump, or it may be done piece-by-piece. But all large-system experience shows that it will be done.”
“Where a new system concept or new technology is used, one has to build a system to throw away, for even the best planning is not so omniscient as to get it right the first time.”
“The throw-one-away concept is itself just an acceptance of the fact that as one learns, he changes the design.”
“Initially, old bugs found and solved in previous releases tend to reappear in a new release. New functions of the new release turn out to have defects. These things get shaken out, and all goes well for several months. Then the bug rate begins to climb again.”
'Things are always at their best in the beginning,'' said Pascal. C. S. Lewis has stated it more perceptively:
“That is the key to history. Terrific energy is expended—civilizations are built up—excellent institutions devised; but each time something goes wrong. Some fatal flaw always brings the selfish and cruel people to the top, and then it all slides back into misery and ruin. In fact, the machine conks. It seems to start up all right and runs a few yards, and then it breaks down.”
“First, the essential problem is communication, and individualized tools hamper rather than aid communication. Second, the technology changes when one changes machines or working language, so tool lifetime is short. Finally, it is obviously much more efficient to have common development and maintenance of the general-purpose programming tools.”
“In a very clear 1971 paper, Niklaus Wirth formalized a design procedure which had been used for years by the best programmers.^ Furthermore, his notions, although stated for program design, apply completely to the design of complex systems of programs. The division of system building into architecture, implementation, and realization is an embodiment of these notions; furthermore, each of the architecture, implementa- tion, and realization can be best done by top-down methods.”
“The flow chart is a most thoroughly oversold piece of program documentation. Many programs don't need flow charts at all; few programs need more than a one-page flow chart.”
“Flow charts show the decision structure of a program, which is only one aspect of its structure. They show decision structure rather elegantly when the flow chart is on one page.”