From The Bit Bucket:
(A)Musings on Engineering, Supervision, and Management
Francis W. Porretto
(Writing as The Curmudgeon Emeritus)
Smashwords Edition
Copyright (C) 2010 by Francis W. Porretto
Cover art by Donna Casey (http://DigitalDonna.Com)
Discover other works by Francis W. Porretto at Smashwords.Com
Smashwords Edition License Notice:
Thank you for downloading this free ebook. You are welcome to share it with your friends. This book may be reproduced, copied, and distributed for non-commercial purposes, provided the book remains in its complete original form. If you enjoyed this book, please return to Smashwords to discover other works by this author. Thank you for your support.
==<O>==
Foreword
A sense of responsibility can compel a man to do terrible things. One of the most terrible of all is to assume responsibility for the deeds and foibles of others.
Many years ago, your Curmudgeon was induced to accept a managerial position in his field of software engineering. Mind you, he hadn't applied for the post, nor had he demonstrated any qualities that ought to have predisposed his own management to prefer him for it. They needed a new front-line manager; your Curmudgeon was the most senior and best-thought-of software engineer around; and no one else in the vicinity was anywhere near being qualified for the post. The answer popped out of the slot in their heads whence all half-baked ideas emerge.
Thus, a legend was born. For your Curmudgeon refused to accept the position until he'd been promised a unique privilege: he insisted on remaining a contributing technologist, rather than becoming a pure paper shuffler. His management, loath to court one of the explosions of wrath for which your Curmudgeon is regionally well known, agreed without hesitation.
What follows are essays on a little-understood discipline: the arcane (some would say "black") art of dealing with massive technological challenges while supervising a bunch of intolerable primadonnas and coping with middle and upper managers who think that anything they don't have to do themselves must be easy as pie.
Brace yourself. Pour a big drink. Then assume crash position.
==<O>==
Part One:
Misadventures in the Digital Wilderness
1. Power Tools
Not long ago...well, on a geological timescale, anyway...when he was a wee lad, your Curmudgeon was fascinated by power tools. Drills, circular saws, routers, sanders, pneumatic wrenches, jackhammers, what-have-you. For quite a while it made his parents very nervous. They anticipated a life in work boots for him, which would blast all their dreams of retiring to opulence on his income. (They were greatly relieved when he discovered his love of mathematics, which requires nothing but a chalkboard and time to think, but that's another story.) But just as with all good parents, what they feared most was that he would hurt himself.
Power tools are capable of doing great damage and inflicting great pain. And of course, the less skilled the wielder, the more likely those outcomes are. So it behooves the novice to be extremely careful, to press at the boundaries of his expertise only gingerly, and to stay within screaming distance of emergency help at all times.
The same is true for software tools, though many would refuse to believe it.
Modern programming has become so complex an undertaking, and modern programs so fantastically involute of structure, that without a number of power tools of great sophistication and subtlety we'd get practically nothing done. In your Curmudgeon's world, laboratory simulation, one doesn't consider undertaking even the most mundane tasks unless he has the right power tools staged and ready to hand. These include:
-- Source code control systems,
-- Integrated development environments,
-- Real-time debuggers,
-- Communications interface monitors,
-- Performance profilers,
-- Bug tracking systems.
There are others. Unless he's first marshaled all of these, the modern software engineer is an impuissant creature. With them, he's competent and confident.
Assuming he knows how to use them, of course.
Just as with all that heavy-duty machinery mentioned above, inept use of a software power tool can do great damage and inflict great pain. It's not the same variety of damage or pain, but the effects can be just as agonizing, and last just as long.
Source code control is universally considered the indispensable necessity of all serious programming today. Not to practice it invites all manner of horrors, not the least of which is the grimace of horror one will get from the prospective customer who finds out. But source code control is also a leash, and a trap. No two systems are compatible, so he who chooses control system X has locked himself into it for a long time to come. Almost all source code control systems require database support, and no two databases are compatible, either in their storage rubrics or their query schemes. Finally and worst of all, when a source code control system crashes, it often takes one's source code with it.
That happened to your Curmudgeon not long ago. He's still unable to sleep soundly.
Debuggers, particularly of the sort built into IDEs, are a seduction as well. Every debugger ever written changes the execution environment of any application run under it. At the minimum, they enlarge the executable file and alter the executing program's consumption of stack space and CPU time. Many do far worse. Some are even capable of deceiving the user into thinking he's running a program other than what he really is.
Don't tempt your Curmudgeon to tee off about bug trackers. There are...persons in his professional environment who know how to do nothing but fire up the bug tracker and enter a complaint against his group's work. One complaint: the same one, over and over again, that's been on record for half of forever and was fixed a quarter of forever ago.
Clearly, all of these are power tools. They can make the engineer's life a lot more pleasant, and sometimes they do. But if misused, they can turn what might have been enjoyable work on an interesting problem into a trial that approaches torture.
Education in the use of one's tools is a clear requirement. Sadly, most organizations skimp on this sort of training. In a way, this is understandable; it's quite expensive, and some course vendors don't really give value for value. But the alternative can be much worse.
The appropriate allocation of responsibilities is another obvious need. A significant tool, likely to affect the work of a large number of users, should have a "cognizant engineer," whose responsibilities explicitly include the mastery of the tool and the provision of advanced services to users less intimate with it. His other burdening should never be so severe that he's tempted to stint that aspect of his job. Beyond that, it's a good idea to rotate cognizant engineers away from such responsibilities after a reasonable time, say one to two years, the better to keep them fresh and to promote the general competence of the staff.
Beyond these obvious desiderata lie still others:
-- The development of practices and guidelines that are conscious of the tool, its powers, and its limitations;
-- Care in scheduling maintenance and upgrades, so that users will not be inconvenienced by down time or blindsided by changes in the middle of sensitive work;
-- Regular assessments of the tool's actual impact on the work of the shop, and comparison of that impact to the benefits that were expected from it.
-- ...And last, but certainly not least: a conditioning session for management, during which all supervisors and middle managers are required to scream ten thousand times at the tops of their lungs that "There are no silver bullets."
Allow your Curmudgeon to repeat that with just a tad more emphasis:
THERE ARE NO SILVER BULLETS!
There is no tool, nor can there ever be one, that will compensate for an incompetent or underprepared staff. Power tools are at their most dangerous in the hands of the inept and unready. They acquire even more destructive power when managers responsible for developing schedules and making commitments focus entirely on the tool and cease to see the user holding it.
The modern software development tool should be regarded in exactly the same way as the computer on which it runs, and for that matter in the same way as a power drill or circular saw: as capital equipment to whose powers the engineer must be properly schooled and accustomed. Its power to create is exactly equal to its power to destroy, and it's eternally ready to do either. One would think that these maxims and their implications would be too obvious to require commentary, but experience shows all too clearly that it's not that way.
==<O>==
2. Rules For Tools
Digital engineering is an occupation with various echelons. At the top of the pyramid are those who create brand-new fundamental technologies for use by others: microprocessors, operating systems, compilers, and so forth. At the second level are engineers who devise "macro-components:" for example, board-level circuits with broad but definite uses such as disk controllers or communications interfaces, or "horizontal applications" such as database programs and word processors. At the third level, where the bulk of engineering occurs, are those who use the products of the first and second echelons to solve problems specific to their employers. At the fourth level are support-function engineers such as network administrators, quality-assurance personnel, and customer-support specialists.
The levels are not equal in prestige; no, not nearly. The first level commands the greatest respect by far. Then comes the second, and well below that the third, and then the lowly fourth. Needless to say, every young engineer aspires to rise as far up the pyramid as he possibly can. But each step upward requires greater mastery of a challenging discipline, and greater dedication to its ideals.
This is a case of the 1-10-100 Rule in action. The top level's creations are utterly fundamental to everything below it; mistakes made at that level are capable of wreaking devastation on a gigantic scale. A good example is Intel's historically famous gaffe in misimplementing floating point arithmetic in its original 386-series processors. But of course, second-level engineers can make mistakes as well; theirs are merely more limited in scope. The stumbles of third and fourth-level engineers can cause some transient damage (and quite a lot of embarrassment), but are far more bearable and recoverable than errors made at the levels above.
Essentially, this is about tools: how good tools can ease the labors of others, and how bad ones can leave a trough of destruction both wide and deep. Those who work at any level are engaged in the production of tools and tool systems for use at the levels below. The engineer with ambition to rise through the levels must become ever more attuned to the Rules for Tools, which are graven into the laws of reality as they apply to collaborative human endeavor.
***
Rule 1: A tool is never an end in itself.
Tools are capital goods, not consumer goods. As the old saying goes, people buy drills, but they want holes. Accordingly, a tool must not demand too much of its user; it must accommodate him and his preferences to the maximum degree possible given its core function. A tool that compels its user to adhere to a narrow and unnatural methodology that forecloses his preferred ways of doing things, or makes them too expensive to be borne, is a tool destined for disuse.
***
Rule 2: Within an adequately specified context, the actions of a tool must be completely predictable.
Who would use a machine that takes uniform input and produces non-uniform output? Output that varies eccentrically, according to rules that no one could deduce? Surely not a firm that wants to produce a product on which its customers can rely. A tool must insulate itself against the influence of anything that's not explicitly part of its input and its operating parameters. We may call this the "Black Box" rule: the tool must be functionally ignorant of anything but its specified interfaces.
By corollary, if the surrounding conditions can alter the operation of the tool, or the quality of its output, these possibilities must be delineated in detail, with maximum precision. A toolmaker cannot assume his customers will know anything about his tool, its powers, or its limitations beyond what he himself tells them.
***
Rule 3: A tool's interfaces and usage must be documented with maximum specificity.
When you create a tool, you acquire a responsibility toward him who will use it: a responsibility for providing him with all the relevant information about what goes in, what comes out, and how to control its operation. If there are other constraints on the tool's use or operating context, these must be documented as well -- and without relying on Finagle's Constant. (Scroll to the bottom of this excellent compilation of engineers' saddest wisdoms for the definition.)
Engineers hate this part of their jobs. It requires them to write in English, a language well-suited to human beings speaking of human things, and ill-suited to anything that requires exactitude. Tools, while they're intended to be used by humans, are never as adaptable as humans; nor do most tools automatically compensate for changes in the skill with which they're wielded. The latent ambiguities in even the most carefully constructed description of the proper use of a tool can drive an engineer crazy. Some of these become serial killers; the rest, politicians.
***
Rule 4: If a tool can be misused or mis-customized by the user, his missteps and mishaps in doing so are as much your problem as his.
Users can be contrary about the tools they acquire. They often refuse to read the instructions before attempting to use them, or fail to respect the tool's requirements, and so precipitate disaster upon themselves. This is deplorable. It's wholly the user's fault -- which is entirely irrelevant to what will happen to your reputation, and your employer's, if it should happen too often. What you tell the user about the tool must include as many "thou shalt nots" as are necessary to steer him away from the worst possible foot-shooting scenarios. Such warnings should be printed in letters of fire, inescapably upon the tool itself, or if that's not practical, upon its packaging in such a fashion that the user cannot avoid seeing and reading them. Your company's reputation, justly or unjustly, may depend on how thoroughly and how well you do this.
A truly funny example of this comes from integrated-circuit process engineering: an example of the hazards of using "overloaded" terminology for a highly specialized tool. The tool in question was an "oven:" a heating system designed to finish-bake the packaging in which integrated circuits are enclosed. Such "ovens" are precision instruments that must be kept scrupulously free from contamination, especially by organic volatiles. A friend tells of a rash of ruined chips produced by one such "oven," which, despite being brand new and carefully cleaned and inspected each morning, turned up heavily contaminated at the start of every new day.
It turned out that there was a fellow on the night shift -- not an engineer, thank God -- who saw no reason to limit the uses of the "oven" to baking plastics and ceramics. He was particularly fond of tuna and cheddar cheese on whole wheat, gently warmed. No, he wasn't fired; the company's management had a sense for the absurdity of the situation. He was told in no uncertain terms to stick to the microwaves in the cafeteria henceforward, no matter how long the lines might get -- and the vendor of the "oven" was severely chastened.
***
Rule 5: No tool is right for everything; no tool must be represented as a panacea.
It's well established by experience that the most reliable tools of all sorts are those that do one thing well, and nothing else. The more jobs a tool is intended to do, the higher the probability that it will do none of them well. Even if it does do them all adequately, it's likely to be more expensive, more difficult to use and maintain, and generally clunkier than separate tools, one per requirement, would be.
Yes, there are a few exceptions: "Swiss army knife" type tools that can perform adequately at more than one task. But in truth, such tools are usually backup systems, best employed when the dedicated one-function tool isn't ready to hand. The hardware salesman who tells the aspiring home mechanic that an adjustable wrench and a Leatherman multi-tool are all he'll ever need is committing a crime against his inexperienced customer, no matter what miracles a truly skilled practitioner has coaxed from such a small set of resources in the past.
You'll save yourself a lot of support calls and a great deal of vilification, and protect your company from being tarred as "bad business," by exercising modesty about the applications to which your tool can and should be put.
***
These are the primary Rules for Tools. There are others of lesser import, but without a complete mastery of the ones above, and a good grip on how they apply to his technological specialty, an engineer cannot qualify as a toolmaker and tool-supplier worthy of the name.
Do you have any humorous examples of bad toolmaking or tool-vending practices to tell us about, Gentle Reader?
==<O>==
3. Overreach
Some years ago, the United States Department of Defense resolved to shore up one of its most persistent and annoying weaknesses. The weakness was a form of sectarianism, propped up by key elements in the five services. Each of those services had pledged fealty to a particular religious lexicon, which was proprietary to it. In consequence, their priests were thwarted from working on one another's doctrinal problems by differences among their premises and vocabularies.
Your Curmudgeon will forgive you for muttering "What the hell is he talking about?" The above pattern actually covers a spread of military subjects, but the one closest to the rancid place in your Curmudgeon's heart where he stores his schadenfreude is military programming languages.
As we entered the 1970s, the five services all had preferred programming languages for the computers embedded in their weapons systems, and no two were alike. The Navy favored a language called CMS-2M. The Air Force adopted JOVIAL. The Army made heavy use of FORTRAN-IV. And so forth.
Noteworthily, these languages were either proprietary to the services themselves, or were "on the outs" in the wider commercial marketplace. That made it very difficult to hire persons knowledgeable in them, which increased the costs of development and maintenance for military computer-based systems. As computers became ever more important in military systems, the DoD decided that it had to straighten out the mess.
The result was the Ada programming language and environment, one of the most futile, wasteful boondoggles in technological history. The designers of Ada resolved to cover each and every necessity a programmer might ever require, every convenience he might ever request, in a specification that would free military programmers from having to learn anything except Ada, now and forever. Their rationale was that the common knowledge base of military programmers should enable any of them to work on any program developed for any application -- that they ought never again to have to climb a technological learning curve in addition to the domain learning curve for the problem they faced. Beyond that, a single language and environment would mean a single standard for program correctness, which held out the tantalizing possibility of automatic program verification. Finally, with all five services making use of this one scheme, they imagined that third-party vendors of development tools, libraries, and add-ons would flock to the massive new market and flood it with useful things.
Utopian dreams are no more realizable in the world of technology than in the world of politics. Indeed, any engineer worth his salary will tell you that efficiency and economy are best served by designing the narrowest possible solution to one's problem. But the DoD's planners could not, or would not, grasp that concept. A sufficiently rigorous, extensive, and definitive specification, enforced by the Pentagon's notoriously humorless procurement bureaucracy, would compel technology vendors to conform, thereby reducing military support difficulties and allowing the cross-application of service specialists to one another's most pressing problems.
The official specification for the Ada programming language and environment was first introduced to the world in May 1979. The DoD spent the next twenty years trying to get defense contractors and technology vendors to use it. In 1998, it admitted defeat and closed its Ada advocacy agency.
Why did it fail?
The DoD had devoted immense resources to the study of previous computer languages as expressive technologies, but had given no thought whatsoever to why some had prevailed in the marketplace while others had languished in disuse. Ironically, the two it chose as the most suitable bases for Ada, IBM's PL/I and Niklaus Wirth's Pascal, were two of the least successful languages ever devised, in terms of user adoption. PL/I failed because it was a "garbage can" language: its design committee threw every imaginable feature, regardless of how likely it was to be used, into its definition. Pascal failed because it was designed specifically as an algorithm theorist's teaching language: it was top-heavy on programmer constraints, and lacked the majority of the conveniences practical programmers need to perform practical tasks. Ada exhibited both these flaws, and many others.
Today, Ada survives at a very few shops, while the procurement bureaucracy busily tries to effect its demise in favor of C++. It's impossible to hire anyone who knows it, the range of available tools that will cooperate with it is tiny, and each such tool is an order of magnitude more expensive than comparable items from the commercial world. Thus, it has achieved the exact reverse of its dreams.
***
Ada is a classic case of overreach: the struggle to reach a goal that's innately impossible of attainment given the circumstances and resources at hand. Indeed, quite a number of persons said so at the outset, only to be ignored. The goal glittered too brightly. And of course, as the shape of the inevitable conclusion became clear, the reluctance to admit defeat kept the Pentagon throwing good money after bad for a long time.
Overreach is, of course, to be avoided. The problem is foreseeing it.
The lure of an impossible goal can be tremendously strong. In that strength lies half the overreach temptation: the beauty of the prize causes us not to think through obstacles that suggest that it might be unattainable. That aspect of the problem can be addressed by cultivating a healthy skepticism about technological promised lands and silver bullets. "If it sounds too good to be true, it probably is." The great Fred Brooks, author of The Mythical Man-Month, wrote a somber reflection on this very subject, titled "No Silver Bullet," just a few years ago.
The rest of the lure of an overreach goal is powered by less noble motives: the desire to prevail over others, the yearning to prove that one is the best of the best, irritation with the limits imposed by technology and human fallibility, and so forth. All of these matter individually to some extent; when they combine, their power is greater than the simple sum of their parts, as your Curmudgeon has often had the opportunity to observe.
A recent opportunity was particularly poignant. Your Curmudgeon is employed on a program in which another company, hence to be referred to as The Other Company, is collaborating. The Other Company has a history of doing naughty things to its collaborators, which leads your Curmudgeon to wonder why anyone is willing to work with it.
Well, The Other Company has more than one itching powder in its formulary. Recently, it's tried to impose a networking standard upon one of your Curmudgeon's laboratory simulators, supposedly for our mutual benefit. The proposed standard is very hard to meet, has no bearing whatsoever upon the application for which the simulator is designed and to which it will be put, and involves the use of a completely undocumented and unsupported component devised by one of The Other Company's previous collaborators -- a Finnish firm. Your Curmudgeon deployed his considerable suavity and elusiveness to escape this noose, but the demands became more insistent with every week.
At last, The Other Company pulled the nukes: it dispatched its high expert on this component and its implementation to your Curmudgeon's lab to make the case for it in person. That this expert is a beautiful, sweet-natured, and extremely insistent young woman was probably a mere coincidence.
Your Curmudgeon summoned all his savoir-faire while he listened to the young lady's pitch. He tried his best to keep an open mind about the whole matter. Still, at the end of the session, he remained convinced that this standard and its associated technology was one he'd flee the country to avoid. But he couldn't bring himself to say that to the aforementioned beautiful, sweet-natured, and extremely insistent young expert. Again, probably just a coincidence.
Shortly thereafter, The Other Company's Head Software Leg-Breaker called to ask how your Curmudgeon is doing with his pet technology. He had raised expectations from his expert's site visit. Your Curmudgeon rehearsed his evasions for several days, and ultimately sent the fellow away, if not smiling, at least adequately mollified. But what he really wanted to say, and might just say if the scoundrel were to prick him right, is this:
Sir, this is something I would never allow into any design of mine. It's a classic case of overreach. It attempts to solve far too many problems, and as a result it does poorly at all of them. Worse yet, none of those problems bear any resemblance to the ones I face. Therefore, I've decided that your standard and your component are unsuitable for use here, and I reject both with prejudice.
Watch this space. Or the newspapers.
==<O>==
4. Losing The Race
A highly regarded colleague was recently blind-sided by a mandated upgrade of his development toolchain. (And if you understood that sentence, you're part of a tiny minority.)
Few of us in the engineering world work on fundamental technologies using nothing but our brains and our mechanical pencils. Even "first echelon" technology vendors -- producers of chips, compilers and operating systems -- make extensive use of existing technologies, and are critically dependent on them. Those products become components in the products of "second echelon" companies that make widely applicable systems from them, usually with the assistance of computer-aided-design systems from other vendors. Finally, we in the "third echelon," who have specific applications in mind, choose among the offerings of the second echelon and put them to work. (After us, of course, comes the hapless end user, who mostly has to trust in God.)
Among third echelon software developers, the term "toolchain" refers to all the first and second-echelon technology upon which our work depends:
-- The computers;
-- The operating systems they run;
-- The compilers and linkers we use to create our programs;
-- The libraries of existing routines we leverage to obviate the reinvention of basic algorithms;
-- The debuggers, profilers, and postmortem analysis programs we use to figure out where we failed, and why.
Sounds as if we have an embarrassment of development riches to play with, doesn't it? But the reality is a bit different.
The makers of first and second echelon products are forever striving to innovate, to reduce the cost of their products, to increase their functionality and reliability, to lengthen their mean time between failures (MTBF) and to shorten their mean time to repair (MTTR). Capitalism is like that; even if your customers haven't demanded anything more from you than you're already providing them, you can always feel your competitors' breath on your neck. But there's a funny thing about those innovations and upgrades: because any company will have limited ability to produce new products while supporting old ones, it will often freeze all work on an old version of a product when introducing its successor. There's worse: Sometimes the old version is pulled from the product line, all support for it terminated. So there's no guarantee that version X of product Z will remain useful, or available, once version X+1 is ready for distribution.
That isn't always a harbinger of approaching disaster. Sometimes an upgrade is pure progress. If version X+1 is a perfectly implemented proper superset of version X, entirely backward compatible with all version X add-ons, plug-ins, and parasitic products, and if it consumes no more resources of any sort than version X did under any circumstances, then there's no conceivable danger from it. Happens all the time, right?
Careful, now. Men have died from uncontrollable laughter. Straighten up and try to breathe normally.
Upgrading a key component of your toolchain can completely disorganize a development project of great size. Yet failing to upgrade can cost you in other ways, especially in performance, support, and the provision of badly needed third-party components. The resulting pressures can make you sympathize with the villain-businessmen in Atlas Shrugged, who wanted to freeze all innovation so that they'd never again have to decide between buying a new item at the risk of losing their shirts on it, or not buying it at the risk of losing their market to a competitor who did.
Your Curmudgeon's colleague, whom we shall call Smith, was placed in that situation by his single-board computer vendor. That vendor had decided to cease supporting the use of its board under Smith's version of yet another vendor's operating system. The board vendor compelled Smith to choose between searching for another single-board computer and upgrading to a later version of the operating system. Smith chose the latter. After all, the OS came from an industry leader. It was as well proved out as any product in its category. There'd been no stated changes to the application interfaces. Ought to be a transparent transition, right?
Amazing how close uncontrollable laughter is to uncontrollable tears, isn't it?
Smith is nearing apoplexy. The new version of the OS embeds a "race condition." That is, its exact behavior depends upon which of two independent processes finishes first. It's in the nature of those processes and that OS that their order of completion cannot be made deterministic. When process A finishes first, all is well...but when process B finishes first, all Hell breaks loose.
In a devilish irony, the OS vendor has closed its support offices for "environmental upgrades." Smith has no control over the situation and must wait for those offices to reopen. Yet there are people queued three deep around his desk, demanding to know what he did and how to fix it. Not all of them are managers; some are engineers, and should be expected to know better. Just now, Smith is a perfect protagonist for one of those "want to get away?" commercials Southwest Airlines has been running.
Obviously, this can't be laid at Smith's door, though some in management above his head will certainly try. Worse, it's possible that the OS vendor knows nothing about it either, and will have to embark on an extensive course of debugging to unravel it. But whatever might come of it, it's a powerful lesson to those who undertake upgrades, or command others to undertake them, in the blithe expectation that nothing can go wrong.
==<O>==
5. Giving In
"You young folks don't know how lucky you are. Why, when I was your age, a byte only had two bits -- and they were both ones! We had to program in Roman numerals. You wouldn't believe the rounding errors that caused. Ah, but the men were men in those days..." [all too frequently from your Curmudgeon. Just ask his subordinates.]
Yes, your Curmudgeon is an old fart. A "dinosaur." A relic of an era long gone and best forgotten. But he hasn't forgotten it yet. ("More's the pity," mutter his subordinates.)
Much has changed in the computer industry over the 37 years your Curmudgeon has made it his trade. Objectively, the technological changes have all been for the better. The computers are faster, smaller, more capacious, more robust, consume less power, require less stringent environmental conditions, and are more attractive on or beside your desk. Software is more capable, more user-oriented, better armored against hostile forces and bad input, less likely to crash or destroy data stores, requires less training to use, and more amenable to end-user customization.
But all things have their price. One price of the advances has been the loss of certain sorts of fundamental knowledge by the practitioners in this field. For example, there was once a large body of knowledge all programmers possessed, deemed quite valuable at the time, about algorithms for sorting large amounts of data in a computer with little random-access memory. With the crash in the cost of memory, those algorithms have largely fallen into obscurity. Therefore, a practitioner who discovers that he needs to tackle that chore will approach it unprepared. Similar things could be said about many other kinds of problems, and solutions to them, that were once common, but which technological advances have made rare.
And that isn't the worst of it. Some years ago, it was your Curmudgeon's duty to assist a young graduate student, who had an "intern" position with your Curmudgeon's employer, in getting to work and back each day. One day while we were in transit, this intern, whom we shall call Miss Smith, stunned your Curmudgeon by saying with no trace of embarrassment that she never could understand the difference between disk storage and RAM, or why it was important.
Yes, you read that right. Your Curmudgeon will wait while you unswallow your tongues.
Mind you, Miss Smith was quite intelligent, on the verge of receiving a Master's degree in Computer Science. She was near to completing a major, much needed transformation of our employer's extensive documentation database. But her education in Computer Science had exposed her only to abstract algorithmic theory, and to interpretive tools such as Visual Basic, Access, and Excel. She had never had to run a compiler or linkage editor. She had never had to debug a program interactively. She didn't know what "assembly language" is. In short, she had never had to grapple with the physical reality underneath the virtual world maintained by her interpretive tools.
Yet Miss Smith's skills with those tools were considerable and quite valuable. Your Curmudgeon has no doubt that she received her Master's degree, and went on to become someone's well-paid employee, on the strength of what she knew.
At the time of the conversation mentioned above, your Curmudgeon went into a great, gesture-filled, loathsomely detailed presentation on the differences between RAM and offline storage, why each was necessary and neither was sufficient, and what the divergence between the two could mean according to circumstances. It took the whole of an hour's ride, and he wasn't nearly finished when Miss Smith wished him a good evening, stepped gracefully out of his car, and fled screaming in terror for her dorm room. To this day, he can't be sure that she grasped any fraction of what he said...or, in all candor, whether it would have mattered if she hadn't.
It was possible for Miss Smith to get by without the knowledge under discussion because the tools with which she worked made it unnecessary. Whether it will ever become necessary is questionable; indeed, it becomes less and less likely as time passes and developers' tools increase further in power.
A bit closer to home, one of the projects your Curmudgeon enjoyed most was one in which he had to write a very extensive microprogram for a three-way-parallel, non-Von Neumann processor. It was a bit-slice design, intended to provide a MIL-STD-1553 communications interface to a DEC VAX superminicomputer, and for that application it was utterly irreplaceable. (And if absolutely none of those designations was meaningful to you, well, that's part of what this essay is about.) Your Curmudgeon spent many months on the project -- it required the mastery of two different knowledge domains he'd never needed before -- and was quite proud of the result, which was an order of magnitude improvement on all previous solutions to the problem. When his work was mothballed eighteen months later, at the closure of the enveloping development program, he mourned.
The development of that microprogram was what was once called "bare metal" programming. There were no development supports worth mentioning except for a very primitive microassembler. Debugging was entirely trial-and-error, without even the ability to output print statements to a console. Even the documentation of the board design was very sketchy, barely enough for a determined mind to puzzle out what he needed to know. That's what made it a challenge, and what made your Curmudgeon's achievement a source of pride.
But that program is irrelevant today. Not only has the board it addressed gone out of production -- indeed, the computer it addressed hasn't been made for more than a decade -- but there are infinitely better packaged solutions today that are completely self-configuring and are accompanied by convenience software of great power and elegance. The industry has moved on.
That's what industries do. It's what we expect of them.
Your Curmudgeon has confronted the need to surrender to such progress many times in his years. However great the joy he'd taken in the exercise of his dinosaur's obsolescent bit-twiddling powers, when someone else's advances have made them unnecessary, he's resisted a bit, and grumbled more than a bit, but eventually he's always given in.
Giving in to an advance isn't regress. One's knowledge, just like one's machines, ages and becomes obsolete for immediate practical purposes. Programming with a jack panel gave way to programming in hexadecimal, and then to programming in assembly languages, and then to programming in problem-oriented and high-level languages, and then to programming with the assistance of an "application wizard." Debugging with a front panel gave way to debugging in hexadecimal, and then to debugging in disassembled machine language, and then to debugging in source code with full access to all statements and variables as they were coded. Polyphase merges that required multiple tape drives gave way to hash-table-driven on-disk sorts and "bucket" sorts, and then to indexed-sequential access methods that guarantee that data will never go "out of sorts" in the first place. No doubt some of our current techniques are falling by the wayside as the clock ticks, to be replaced by approaches and conveniences we can hardly imagine.
Yet giving in can be very hard. It stabs at our self-regard to admit that our hard-won expertise might have become permanently irrelevant...that younger folks already conversant with the new tools might have stolen a march on us. Alongside that, we fear. We fear that it's not just our knowledge that's becoming obsolete, but the ability it allows us to exercise, for a profit, on others' behalf.
But time marches on.
Your Curmudgeon has had to do a lot of giving in, these past few years. The utter and complete triumph of Windows-based microcomputers over all other systems, for all but a vanishingly small set of applications, compelled him to give in on his love of VAX/VMS and UNIX. The ascendancy of graphical-user-interface programs, so beloved by users for their convenience and self-explanatory nature, compelled him to give in on his old command-interpretive techniques. The emergence of powerful application development assistants such as Visual C++, the Microsoft Foundation Classes, and Access, which obviate most of the low-level fiddling that used to dominate program construction, compelled him to give in on his preference for "doing it all himself"...especially given the complexity of Windows programming, and the demands of his customer communities for fast results.
By giving in over and over, your Curmudgeon has retained a fingernail grip on competence and commercial viability. But one must give in on more than just the obsolescence of one's technical knowledge. It won't be much longer, perhaps only a decade or so, before he'll have to admit that the learning curves presented by the acceleration of technology have grown too steep for his aged brain. That surrender is a fearsome prospect indeed, for it heralds the approach of another, more comprehensive surrender that all men fear to face: the descent into irrelevance, dependence, and the final decline.
To the best of your Curmudgeon's knowledge, Microsoft isn't working on anything that will help with that.
==<O>==
6. Unforgivable Sins
Your Curmudgeon likes to speak of the modern engineer as an ethical force, but he must admit that not all his colleagues in software hew to standards of which he approves.
There are several sins, or more precisely categories of sin, that rise to unforgivable status. Nothing the perpetrator can say, do, or offer the afflicted one will win him absolution. Unfortunately, because software is becoming ever more complex and vendors are growing impatient with the delays and costs involved in both development and testing, these offenses are becoming more frequent by the day.
-- Losing The Customer’s Data.
Under no circumstances may a vendor destroy, corrupt, or otherwise compromise a customer’s data. Neither shrink-wrap nor custom software developers will be forgiven for such a crime. This is especially vile when committed by data storage and management programs. When the offender is a version-control system or an archiving program, it’s Satanic.
A vendor of a popular version-control system recently corrupted over six hundred of your Curmudgeon’s development files. It was possible to recover only because the archiving system on his network had captured all those files the night before. But during the recovery procedure, the SysAdmin let slip that the archiving system had recently turned itself off for several weeks, without any notification to anyone, and had been discovered dormant quite by accident a few days before.
-- Destroying Backward Compatibility.
In the world of office applications, the major vendors for some years competed with one another by adding ever more exotic features to their spreadsheet programs / word processors / database programs / what have you. These bits of exotica often required a modification to the file formats used to store the relevant objects. Unfortunately, those modifications were seldom properly designed; even spreadsheets / documents / databases / et cetera that didn’t use the newly added features could not, by any known method, be back-converted to the earlier file format(s) so earlier versions of the relevant program could make use of them. If your document had been produced by SuperScribbler V12.7, then only SuperScribbler V12.7 and its notional successors could even open it, to the extreme dismay of your colleagues running SuperScribbler V12.6.
This is bad for several reasons. New versions of an application, especially a complex shrink-wrapped program, will often contain new bugs. Sometimes those bugs don’t surface until a significant amount of work has been committed to the new version. But the lack of a path backward binds you to the bugs until another upgrade has been announced. Equally important, if it’s impossible to share your work products with anyone who’s not quite as up to date as you, your ability to collaborate with others is radically reduced. Also overlooked far too often is this consideration: new versions that require more RAM, disk space, or processor power than their predecessors might not run on all the computers their predecessors will run on; thus, a software upgrade by one member of a working group can force expensive and hazardous hardware upgrades on everyone else in the group.
-- Wild Alterations To The User Interface.
Few things are as difficult for your Curmudgeon to understand as radical changes to an existing user interface. The customer learns and gets comfortable with a program’s user interface at his expense. He invests in it. When the vendor arbitrarily changes that interface in a dramatic fashion, he forces his customers to write off that investment. If there are many persons involved with the program, the cost of reacquaintance can be high.
Sometimes, the expense is greater than merely climbing a new learning curve. When an important development tool completely transforms its user interface, rendering it unrecognizable to earlier generations of users -- why yes, your Curmudgeon does have Microsoft Visual Studio in mind! – the perceived human cost of upgrading can appear so high that the customer base pins itself to the superseded version. In effect, the vendor has persuaded a large fraction of its customer base to cut itself off from ongoing support, as superseded versions are rarely supported for long after their successors are introduced.
-- Invalidating An Existing API.
This might be worst of all. An API – “application programming interface,” for those who don’t recognize the acronym – should be considered sacrosanct. That was the motivation behind the Microsoft COM and DCOM standards, the CORBA standard, and a number of similar application interface patterns. But a surprising number of vendors regard their allegiance to such meta-standards as revocable by either party at any time.
When the API is that of a device driver -- a program that communicates with and manages a peripheral device at the behest of the operating system or an application – the offense becomes capital. Your Curmudgeon is fortunate in that he possesses the skills required to write and debug device drivers, but this is a blessing shared by few. When a vendor arbitrarily makes changes to his driver’s API, he invalidates all programs tied to the earlier versions, and forces the hapless customer to retest and revalidate any and all code that required them. In a sophisticated development environment, this can be quite as costly as a disk crash.
==<O>==
7. Is It Time To Despair?
Jeff Medcalf of Caerdroia, one of the Blogosphere's most intelligent participants and long a Curmudgeonly favorite, has penned an insightful, if gloomy, piece on the inherent limitations of software engineering. Here's the core of his thesis:
I had an epiphany a few weeks ago, wherein I realized that a large common element of many of the design problems being encountered in projects I work with is not that the developers are not good at figuring out how to correctly factor business logic, but that business logic is not an object or a characteristic of an object, so expressing it as objects is fundamentally wrong. Object orientation is just too simple to provide the correct abstractions beyond the model layer and the display-interaction part of the GUI (which is inherently physical). It cannot express business logic (processes, procedures, constraints, timing sequences, coupled rule sets and the like) appropriately.
Every word of this is gospel truth...yet in point of actual, factual, unsatisfactual practice, the problem is even worse than that.
The object-orientation approach to improving software development was a red herring from the start.
Mind you, this is not to say that there's a logical problem with the principles of object-oriented design and programming. Those principles -- encapsulation; inheritance; polymorphism -- are all just dandy if they're applied with adequate knowledge and rigorous attention to the domain of interest. But as always, the most important word in the preceding sentence is "if."
Object-orientation, which your Curmudgeon will call OO henceforward to save precious pixels, is a discipline imposed on the programmer's choice and use of tools. That discipline's principal motivations are:
-- To promote the creation of reusable solutions to common problems;
-- To control the proliferation of interfaces within large programs.
Neither of these goals is unworthy...but neither of them sticks its nose out from under the software-theory blanket.
Here, in one sentence, is the heart of any and every problem ever encountered in software engineering toward any end:
The software engineer always has two domains to conquer, and is expert in at most one of them.
The first domain, in which the SE has some (assumed) competence, is programming computers. He knows a language or two, and some of the basic principles of programming. That qualifies him as a software engineer, at least by bean counters' definitions. But the second domain, that of the application whose problems are to be solved or whose processes are to be automated, is one in which he is typically far from competent. To do anything useful for his customer, the presumed expert in the application domain, he must learn what his customer knows about his needs. OO does not help with this. Indeed, by promoting the notion that large chunks of the solution to the SE's current problem might be extracted from his solutions to previous problems, it might even impede it. The application domain is prior and superior to any tools used to approach it.
This is the major and ineradicable challenge of software engineering, and will remain so until the computers grow weary and exterminate us all.
Consider how different this is from similar tool-oriented trades. The commercial electrician isn't expected to work on anything but electrical circuits that vary very slightly from one to the next. The auto mechanic isn't expected to work on anything but cars, and in many cases (e.g., dealers' mechanics) is able to restrict his problem space to only a certain marque or model. The software engineer is expected to turn tricks with any sort of data or information-processing problem whatsoever.
It's a formula for disaster, and disaster is what we've got.
When Jeff says that "[software objects] cannot express business logic (processes, procedures, constraints, timing sequences, coupled rule sets and the like) appropriately," he's expressing a special case of a general rule: software objects cannot express anything but specific properties pertaining to software operations. Software operations are inherently abstract. To be usable, they must be customized to the application of current interest -- and there we are back in the application domain, where the software engineer is more or less incompetent.
In the world of military software engineering, where your Curmudgeon earns his daily bread, we approach the problem by training application-domain experts. Sometimes these persons carry special titles, such as "requirements engineer," or "technical adviser." Their job is to understand the application as if they were the customers and ultimate users of our wares. But they do no programming; instead, they act as surrogate customers, available to the software cadre to guide the design and development of the product. The advantage, of course, is that they're "ours," usually have some engineering background, and available to us at need, whereas the customer has his own concerns, knows nothing about engineering, and can be maddeningly remote.
Yeah, right. What routinely happens is that these "requirements engineers," by virtue of their intimacy with the application domain, routinely get shunted off into product planning, customer-liaison and sales work, which makes them unavailable to the software engineers. After all, they grasp the customer's needs and speak the customer's language; who better to deal with him directly? And we simply must protect such a person's time from consumption by mere grubby programmers; after all, he's got a plane to catch!
And so it goes. And so it will go unto the end of the world, barring one conceivable but unlikely development: the fostering of application-oriented specialties among software engineers, to a far greater degree than has yet been embraced.
Don't expect that. It means that, rather than thinking of oneself as a software engineer, one must accept that one is a personnel systems software engineer, or an AEGIS-compliance software engineer, or a power distribution and load-balancing software engineer. Your typical SE isn't going to go for that. The great appeal of our field, to most of us, is its breadth of applicability. We like to preen and say that if it can be done with a computer, we can do it...even if we have no idea where we'd begin. To deliberately narrow one's claim of expertise in such a fashion is all but unthinkable. Did it make those old COBOL guys a ton of money solving Y2K problems in 1998 and 1999? "Yes, but where are they now?" we would reply.
The problem is stiff. Partial solutions are sometimes available; for example, your Curmudgeon's group does mostly military aviation simulations. We're all quite competent in simulation work, and we learn quickly enough to master most challenges within our chosen domain. When we stay within that domain, we're about the best around. But recently, your Curmudgeon tasked a subgroup to tackle a database-intensive numerical analysis problem with significant human-engineering aspects, and shortly thereafter wished he hadn't agreed to do so. He's learned all over again why domain knowledge trumps skill with a computer.
No, it's not yet time to despair. But we could use a lot more humility, and a lot more candor about the steepness of our learning curves and the limitations on our natural abilities, than we commonly deploy.
==<O>==
8. Software Heroes
No, it's not quite as evocative as "war heroes," "gridiron heroes," or even "guitar heroes," but one does what one can. Anyway, who are they? In your estimation, that is.
Your Curmudgeon has his own nominees, of course. A recent adventure reinforced the concept for him with painful sharpness. The stimulus was the release of a radically new version of a simulation your Curmudgeon is responsible for -- so new that it shares not one line of code with its predecessor.
The predecessor version was about 100,000 lines of C++ code, and rather rickety code at that. It's comprehensible stuff, but it's written in a very old style, is overly sensitive to changes in requirements, doesn't exploit many commonalities that could have been used to reduce its size and improve its performance, and has proved difficult for your Curmudgeon to maintain since the original implementer retired a year ago. The new one comes in at about 30,000 lines, all of it your Curmudgeon's handiwork.
But no software artisan replaces an existing program with an entirely new one, intended to replicate its predecessor's functionality at all points, without quite a bit of fear. The new program must be rigorously tested. Tested? Nay, savagely beaten, brutally abused, thrashed to within an inch of its life, before it can get anywhere near the user community.
The implementer must sic truly skilled testers, masters of the art of software abuse, on his baby before he lets it see the light of day. But such testers are rare. The few genuine experts at this discipline work in closely guarded labs and command high salaries. The rest of us have to muddle by on mediocrity.
An experienced engineer will tell you: The implementer's tests of his own work are next to valueless. Modern science doesn't yet know why, but it's been empirically confirmed and reconfirmed that implementers somehow avoid stressing the true weak spots in their programs. It might be conscious, it might be subconscious, and it might be the work of evil telepathic gremlins from Epsilon Eridani, but it is undeniable. A program tested only by its creator will fail catastrophically within its first hour of exposure to a genuine user.
So the smart engineer cultivates the good will of persons who've shown some skill at testing. Unless he's very fortunate, he won't have access to the millennial geniuses of testing mentioned above. But a gaggle of users willing to "beta-test" -- that is, to use his still-damp creation as if it were a production-quality item -- can serve almost as well.
Your Curmudgeon is blessed with such colleagues, and they've served his need admirably well. Jimmy, Mike, thanks is far too pale a word. This grizzled old bit twiddler is deep in your debt.
***
The future of software engineering belongs to the tester.
Software of all sorts has become so complex that your Curmudgeon refrains from buying Version 1.x of anything, no matter how large x might be. Many persons rail at specific companies about cavalier test practices and treating their customers as conscripted beta-testers, but the problem is pandemic, and nowhere intentional. As there's little to no chance that software will get less complex, nor that customers will become more generously disposed toward easily broken early versions, the need for rigor in testing, applied by persons with a true grasp of the discipline, will only grow.
If your Curmudgeon were entering the field rather than on the verge of leaving it, he'd make testing his specialty.
Unfortunately, testing is regarded by many young engineers as a low-prestige post, with much justification. The tester's name doesn't appear on the package; indeed, the tester is seldom publicly linked with the programs he tests in any way. Many companies have used their Quality Assurance departments as "pastures:" places to hide engineers near to retirement who can no longer handle the pressures of development work. The consequences are before us.
This must change. If there's any set of changes more important to the software industry than bringing about routinely rigorous and reliable testing of its products before they're shrink-wrapped, your Curmudgeon can't think of it today, and doesn't expect it to occur to him tomorrow. The only way such a regime could possibly come about is if managements start treating Quality Assurance as a high-prestige assignment, deserving of generous remuneration and public acclaim, so that young engineers of skill and verve will be drawn to it.
Your Curmudgeon regrets to say that that doesn't look too likely in the near term.
***
Skill at testing necessarily includes skill at developing test harnesses. In the world of software, a test harness is a "non-product" program (i.e., never to leave the development shop) used to exercise the "product" program (i.e., the one under development, which persons other than the developers and testers will use). A good test harness will permit a tester to inject any imaginable input into the program being tested, along any of its interfaces, and will capture the program's response for the tester to inspect and ponder. Without a good test harness, the tester is compelled to use ad-hoc methods of stimulating the program under test, and is seldom able to produce a range of inputs or a degree of stress great enough to establish confidence.
Most software engineers resist writing test harnesses. They regard them as "throwaway code," of limited and transient value. Many feel that effort directed other than into the "product" program is effort wasted. This is a severe misconception.
Were your Curmudgeon hiring for a position in his group, given two applicants of apparently equal ability, he would prefer the one that showed some enthusiasm for testing, and in particular for the development of test harness programs. That affinity would even outweigh substantial differences in education and experience, though not in intelligence or initiative.
The new simulation mentioned at the beginning of this post benefited from an elaborate test harness: a program of about equal size and complexity, written specifically to test the new simulation. It is not "throwaway code;" your Curmudgeon did not regard the time and effort he spent on it as an unfortunate diversion from "real work." Should the "product" program ever need to be extended, the test harness will come out of mothballs and be extended alongside it.
Testing never really ends, you see. The testing baton passes from the developer to the Quality Assurance department, then to the beta-group customers, then to the general public...and when a customer manages to break his program, right back to the developer again.
Call it the Circle of Digital Life, from which there's no escape.
==<O>==
9. Adventures In FibreChannel-Land
Download this book for your ebook reader.
(Pages 1-35 show above.)