CHANGE: Planned & Unplanned
Volume 8 of the Quality Software Series
by
Gerald M. Weinberg
SMASHWORDS EDITION
PUBLISHED BY:
Gerald M. Weinberg on Smashwords
CHANGE: Planned & Unplanned
Copyright © 2011 by Gerald M. Weinberg
All rights reserved. Without limiting the rights under copyright reserved above, no part of this publication may be reproduced, stored in or introduced into a retrieval system, or transmitted, in any form, or by any means (electronic, mechanical, photocopying, recording, or otherwise) without the prior written permission of both the copyright owner and the above publisher of this book.
Smashwords Edition License Notes
This ebook is licensed for your personal enjoyment only. This ebook may not be re-sold or given away to other people. If you would like to share this book with another person, please purchase an additional copy for each person you share it with. If you're reading this book and did not purchase it, or it was not purchased for your use only, then you should return to Smashwords.com and purchase your own copy. Thank you for respecting the author's work.
Part I. Planning for the Future Organization
Recognition of the distinction between a stable system and an unstable one is vital for management. The responsibility for improvement of a stable system rests totally on the management. A stable system is one whose performance is predictable. It is reached by removal, one by one, of special causes of trouble…
There are many ways to categorize software organizations. In this series, I've used the concept of software cultural patterns, based on the work of Philip Crosby. The focus of this volume is on moving toward a Pattern 4 (called "Anticipating") culture, assuming that you already have a Pattern 3 (Steering) culture. Every culture changes, of course, but the Anticipating culture is a continuously changing culture that plans its changes to shape the organization it wants. Thus it is concerned with both the process of fostering change and the process of maintaining stability in the face of change.
Most writers on improving software engineering emphasize the essential changes, but neglect the subject of how these changes are to be brought about. These writers seem to assume that all that's required is telling people what to change—but this is the Pattern 2 (Routine) paradigm.
The Pattern 4 paradigm emphasizes process before product—and this emphasis carries over to the process of changing the organization itself. Until you know how to plan for change, you will continue to be frustrated—knowing which things to change, but never seeming able to accomplish them, even with the help of experienced change artists (the subject of Volume 7). If you hope to transform your software engineering organization to Pattern 4, you must master the art of planning for change, and that is the subject of Part I.
Chapter 1. Meta-Planning: Part I. Information
Climate is what you expect. Weather is what you get. - Anonymous
The winds and waves are always on the side of the ablest navigators. - Edward Gibbon
1.1 Start by Meta-Planning
Figure 1-1 gives an overview of strategic planning as practiced by many effective software engineering organizations.
Figure 1-1. The strategic planning process uses both internal and external information to generate visions of both product and process. These visions, in turn, drive the action planning process.
The plan addresses two broad questions for a time interval that is typically three to five years long:
• What products/services will we supply?
• What processes/resources/culture will we need in order to supply them?
These two visions are, of course, interrelated, so the strategic planning process will have to be able to estimate the feasibility of some of the visions, asking:
• Can we build these products with the processes/resources/culture we now have?
If the answer is yes, then the process vision is one of maintaining and perfecting the present process. If the answer is no, then the planners must decide whether to reduce their ambitions or to raise their process vision.
The reason we create these visions of product and process is for them to become inputs to a tactical planning process. Some organizations, however, lose sight of this purpose, and the actual strategic planning process more closely resembles Figure 1-2, which was drawn for me by one of my clients. In this so-called strategic process, planning becomes an end in itself, unconnected with the rest of the organization, and producing lots of filed paper but no action.
Figure 1-2 How strategic planning is actually practiced in some organizations.
Looked at cynically, the process of Figure 1-2 is what you get when some of the big shots are given the chance to play at being "executives." A more realistic interpretation is that these are scared folks, just like you and me, who have been forced by their management to carry out a process for which they have no experience, aptitude, training, or skill.
Managers who do not know how to get useful feedback about business information feel out of control. The only things they have any sense of control over are time and money—hence the popularity of budget and deadline restrictions. No wonder so much drinking goes on. Perhaps the organization is fortunate the "vision" document hides in the files, never to be seen again.
When done well, however, the entire process of strategic planning for change can be regarded as having three major components carried on iteratively to produce successive versions of the plan:
• information gathering, including both observing and ignoring information from the entire organization and all parts of its environment
• problem solving, including systems thinking, negotiating, and translating into action-generating principles
• mechanics, including knowing when, where, and how the planning is done, as well as who is involved, and understanding the difference between tactics and strategy
In Variable (Pattern 1) and Routine (Pattern 2) organizations, strategic planning attempts are almost always a fruitless exercise. Until problems in all three components are cleared up, the essentials needed for a fruitful change strategy planning are simply not available. In these circumstances, you need to bootstrap the planning process. You must restrict the output of your first planning sessions to meta-plans—plans that need to be carried out for the planning process itself before you can do effective planning on the rest of the organization (Figure 1-3).
Figure 1-3. The effectiveness of meta-planning has a major impact on the quality and effectiveness of every stage of the organizational change process.
Some of these meta-plans will concern the quality and quantity of information, and that's mostly a matter of skill at communicating with people. For instance, one of the most powerful meta-plans is a blueprint for training to raise the communication skill level of the planners. Some of the meta-plans will deal with mechanics, such as steps leading to the use of a skilled facilitator. And some of the steps will deal with problem solving, such as learning to think about risks and trade-offs.
The sections of this and the next chapter consider each of these components from the point of view of how they can be managed or mismanaged. These chapters provide a checklist of dangers you need to watch for, and suggests meta-plan actions to initiate when you detect any of those dangers.
1.2 Information Gathering
The information for making strategic decisions needs to come from both inside and outside the organization. The quality of the planning will be supported by the quality of the information. Three types of problems plague the quality of the information used in strategic planning:
• Omission of some essential source, particularly those doing the work
• Dependence upon an unreliable source
• Wallowing in excessive detail, while ignoring relevant details
Let's study each of these problems, to discover their causes and to frame meta-plan actions that might overcome them.
1.2.1 Omission of information
As with any meeting, the success of a planning meeting is 90% determined by the events that take place before everybody gets into the meeting room. Nowhere is this principle more clear than in the case of missing information. Managers may be the smartest people in the organizations, but they're not psychic. Here are some typical problems of missing information, their causes, and suggested actions to remedy them:
Problem: Omitting customers.
Software engineering organizations often fail to consider customers' wishes in their planning process, arguing customers don't know what they want, or else they want everything. Naomi Karten gives an example that would be funny if it weren't tragic:
A company invited me to help them establish a service-level agreement. When I got out there, they explained that this effort was part of a comprehensive IS-wide strategic review and a revamping of their services, policies, etc. They explained that this effort had been given the name The Voice of the Customer, and it had been in progress for more than a year. "Oh," I asked, "and to what extent have customers been involved?" "Not at all," they told me. The Voice of the Customer?
Cause: Though it hides behind blaming, this problem is pure irrelevance. Managers hide out because they feel inadequate when actually forced to communicate and negotiate with customers. Some managers hide because they don't know how to tackle what appears to be a massive and unfamiliar undertaking in which they fear they'll have to give away the store.
Meta-Plan Action: In the strategic meta-plan, specify that customer satisfaction will be measured regularly and systematically. Also specify that customer desires will be updated regularly, using such varied techniques as brainstorming, focus groups, surveys, and questionnaires. Until you've executed this meta-plan, there's not much sense holding your first planning meeting.
The measurement of customer satisfaction need not be overly precise, nor imply that every customer's tiniest whim must be satisfied. It does have to be accurate, however. It also has to be made available to everybody in the development organization, so they can understand their customers and will have the information needed to make sensible decisions on all scales. As my colleague Lynne Nix points out, developers in many organizations don't even know how to use the product they're developing. How could such developers ever make sensible decisions about anything that might affect usability?
Sometimes hearing "the Voice of the Customer" involves removing management-imposed barriers. For example, Norm Kerth observes:
One of my clients won't let their software developers meet their customers because
(1) "They would embarrass us."
(2) "They would make it a boondoggle."
And Phil Fuhrer adds, from his experience,
(3) "They might disclose product plans before they are ready."
(4) "They might make promises that can't be kept."
Problem: Omitting potential customers.
Frequently, these organizations fail to consider potential customers: those people who might be using their products and services, but are not. One anonymous correspondent from a failing software company told me:
At our peak, we had about 1,200 customers for our mainframe package. When we were down to about 600 customers, I suggested we interview some of the customers we had lost to find out what we should do differently. I was told that we didn't want to hear from people who weren't "loyal" to our company. When we got down to 300 customers, I left. Now I'd be surprised if they still had 100.
Cause: Managers do not know, or are afraid to use, any processes to gather information on satisfaction of existing customers, let alone potential customers, so they deny that such information is important.
Meta-Plan Action: In the strategic meta-plan, specify that customer measurement be extended beyond current customers to include former customers and potential customers.
Problem: Omitting overall business strategy.
Even if they completely ignore their customers, managers, you'd logically assume, would not want to ignore their own executives. Thus, they would want the corporate business plans to be part of the planning information base. Yet, software engineering managers are often unable to demonstrate any relationship between their software engineering strategy and the firm's overall business strategy. Indeed, in one organization,
Someone said we should look at the corporate strategic plan. Nobody had one at the meeting, so we took a break while we went to look for one. After thirty minutes, nobody could find a copy, so we resumed our planning session without it. The next day, somebody brought one, but by that time, most of our work was done. Besides, everybody thought it was too thick to read.
Cause: Sometimes the planning process produces a report purporting to show how software engineering relates to the overall corporate strategy—but the report turns out to be a fraudulent assemblage of obfuscating verbiage. In one organization, a secretary is given the job of preparing this document, filling in arbitrary connections according to a formula. This process actually saves executive time since nobody will ever read the document anyway.
Meta-Plan Action: In the strategic meta-plan, specify that departmental plans be reviewed by upper management before they become plans, and that steps be taken in the planning process to ensure this review passes before the process proceeds.
Problem: Omitting any organizational information.
Amazingly, many strategic planning sessions use essentially no information of any kind from within their own organization. A technical lead told me this story:
I'd never been at one of the strategic off-sites, but my boss was sick so I had to take her place. The only thing they used was an org chart, and they jockeyed back and forth, trading people and projects like the National Football League. I was scared to say anything, so my boss lost 7 of 29 people who were working for her. She was really mad at me, but I guess it didn't matter, because I was one of the 7.
Cause: Quite possibly, no useful data have ever been collected. Instead, participants simply give their opinions of how things should be. They don't have any measures of the productive capacity of their organization, nor do they have a clue about the magnitude of the design maintenance debt of existing products they plan to modify.
Meta-Plan Action: In the strategic meta-plan, specify that staff along with the managers—not the managers alone—establish a system of measurement to provide the measures needed for future strategic planning.
Problem: Omitting any outside information.
When it comes to outside sources of information, the track record of these organizations is no better.
Cause: Frequently, they enter the planning process with lots of technology data, but this tends to be supplied by a favored vendor or two. Sometimes the entire executive crew is wafted off to a vendor site for a technology briefing as preparation for the planning process. One of the more enterprising vendors actually "facilitated" the strategic planning at the Country Club, with some of the major decision-making done on the golf course.
Meta-Plan Action: In the strategic meta-plan, specify that staff with managers establish a system of environmental search—scanning what exists in the world outside the organization—to provide the measures needed for future strategic planning. This search is not to be limited to technological information, for technology is only a small part of the total context in which a system lives. Moreover, the sources need to be reliable and unbiased.
Problem: Lack of organizational development knowledge.
As poorly informed about technology as the planners may be, when it comes to knowledge of organizational development possibilities, many software engineering managers are better informed about the performance characteristics of their BMWs. At least when it comes to the state of the economy and their industry, they read Business Week or Time. A fellow consultant told me:
I conducted a telephone survey of the nine managers, one of whom had invited me to facilitate a strategic planning meeting dedicated to upgrading their development organization. Six of them had never heard of the Software Engineering Institute. Two of them had heard of it, but couldn't tell me what it was, or what it did. The only one who knew something about it was the one who invited me. I suggested they do some reading before they had the meeting. They got themselves a different facilitator.
Cause: Many software engineering executives lack training in what structures and processes are possible in organizations. Moreover, since many have spent their careers in a single organization, their experience is limited.
Meta-Plan Action: In the strategic meta-plan, specify training in organizational possibilities for future planning participants. This training should include working in a staff position for a period of time to experience the workers' realities. Also specify participation in planning sessions by some organizational development specialists
A radical alternative solution is to abandon strategic planning for change altogether and adopt the SEI approach whole hog. At the very least, the planning team should be required to read and discuss the SEI's Capability Maturity Model (CMM). Although I disagree with the SEI on many points, and I believe the CMM doesn't cover half of what has to be done, this approach would save a lot of time, and produce a better plan than three-fourths of the strategic change planning efforts I've witnessed. If you can't do it better, why not reuse someone else's work—and be a model for your developers?
Problem: Running sham planning sessions.
Given a lack of valid information, any sort of planning would be a sham; yet it continues to be done in this way, at great cost and disruption to the organization. Such a sham demoralizes workers and discredits their management. As one wag expressed the principal advantage of strategic planning in his organization, "At least it keeps them out of the office so we can get some work done."
Cause: When the planning is strategic, however, the impact is so far in the future that it's easy to hide the emperor's nakedness. When all the alleged planning is done at a luxurious executive retreat, the grunts at the office may suspect it's a sham but, like their managers, have no data to prove their theory.
Meta-Plan Action: In those organizations that do plan effectively, strategic planning sessions are not some sort of executive perquisite, but hard, intelligent work based on meaningful data. Specify that the planners do their homework and come to the sessions informed about the critical areas that will affect their plans. Hold the sessions more openly, and conduct them in slightly more Spartan surroundings than a five-star resort. Another good idea is to require a public display and review of a substantive work product produced by the planning.
1.2.2 Unreliable sources
Unfortunately, simply working hard to gather data is not sufficient, because most of the data that are readily available are of questionable quality.
Problem: Running on "visible figures alone."
Internal reports are not reliable indicators of what's happening in the organization, or if they are reliable, they're irrelevant.
Cause: Deming's Fifth Deadly Disease is "running a company on visible figures alone." He points out that there are some good things like ecstatic customers that cannot be reduced to figures in any regular way. But fear of customers and employees leads managers to hide behind their spreadsheets.
Meta-Plan Action: Use the technique of "management by walking around" prior to planning exercises to validate those information systems that are buried several levels down from the manager. As skilled managers know, a few sample conversations will quickly indicate whether or not the information in formal reports has the meaning it appears to have. Some managers hire consultants to do this, but walking around does the same and more if the managers know how to listen. If they don't, they create great, unregulated disturbances by walking around, and they're better off hiding in their offices.
Problem: Unstable current processes.
The processes being measured are so unstable that measurements have no meaning except to show the instability, as one developer illustrated:
Our test manager reported to the meeting that the bug file had increased from 11,392 to 12,514 since the last monthly meeting. This, he said, was an indicator of progress, because this was the first time since we started measuring that the monthly increase had been less than 10% (It was 9.8%).
Cause: The causes of instability are many. Systems thinking—the ability to reason about nonlinear feedback systems—is needed to root them out.
Meta-Plan Action: In the strategic meta-plan, focus on the steps necessary to stabilize any unstable process before any attempts are made to optimize it. For instance,
• What does it take for measurements to have meaning?
• How can the results of those measurements be fed back into the planning process in a stabilizing way?
• What do we have to do so we can recognize commitments to customers?
• How do we know when projects are finished, rather than just delivered?
Problem: Confusing opinions with data.
Organizations need strategies based on data, not opinion, no matter how the information is gathered. One correspondent offered me the "proof" his management had given him that their process improvement efforts were right on track:
We pay (about $150,000) for an annual survey conducted by an outside consulting firm. This year, the survey showed that the managers thought our software development effort had improved from an average of 3.4 (out of 5) to 3.9.
Cause: People lose sight of the difference between data and opinions about data. A thousand managers thinking that an organization is on track doesn't make it on track.
Meta-Plan Action: In the strategic meta-plan, call for measurement systems that can be replicated and validated, independent of opinion. For instance, projects can be tracked with Public Project Progress Posters (PPPPs) so the open air will cleanse the opinion from the data. If they are properly planned, PPPPs apply to organizational change projects just as well as they do to software development projects.
1.2.3 Wrong level of detail
Even when the available information is reliable, planning sessions slip into trouble by working at the wrong level of detail.
Problem: Wasting time on irrelevant details.
The planning group spends too much time on irrelevant details. How to determine if the details are irrelevant? It's usually as obvious as one correspondent reported to me:
I once watched in awe as a group of managers spent more than three hours of their two-day off-site planning meeting designing a new logo for their process improvement campaign..
Cause: A planning group wallowing in irrelevant details is usually a symptom of something else that's wrong. It may be that the group lacks a sense of the planning process, or is poorly facilitated. It may be that the group has run out of real issues, or has no real data on real issues. Frequently, there is some hidden issue the group is dancing around.
Meta-Plan Action: When you become aware a group you're in is wallowing in detail, comment on the situation. For example, you might say, "I notice we have spent a lot of time on what seem to me to be details. Am I missing the issue here?"
A good process should help eliminate this problem. If you have a good process, the question might be, "Are we following our process now?" or "Where are we in our process right now?"
If the process isn't so good, the question might be, "Is our process working for us right now?" or "Do we need some different process to keep us on track?"
Problem: Comparing apples to horses.
Data from different sources are not of comparable detail. Here are three lines from a spreadsheet labeled "Strategic Plan":
Early line: Estimated workload increase: 30%
Middle line: Average line of code, labor: 48 minutes, 37.2 seconds
Bottom line: Needed increase in person days: 2,397.874.
Cause: The most frequent source of this problem is when historical data are compared with predictions of the future.
Meta-Plan Action: The planning team insists that data be supplied with estimates of variance. For illustration, notice the difference between the measures in each pair:
• Development cost per function point = $700 ± $500 vs. $700 ± $50
• Predicted market share = 17 ± 7% vs. 17 ± 1%
• Cost of process improvement per developer = $1375 ± $900 vs. $1375 ± $90
The large variance on predictions will limit the detail used from other sources. In the strategic meta-plan, specify this estimation of variance for all future measurements.
Problem: Inconsistent attention to detail.
Either every issue is worked to death, or important issues are treated superficially. My correspondent reported that in the same meeting that devoted three hours to a logo, the question of training was dismissed with the comment, "Oh, the training department knows their job."
Cause: Before the planning session, no system of priorities exists, or there is a bogus one such as stating "all issues are priority 1."
Meta-Plan Action: In the strategic meta-plan, specify that future issues will come to the planning session with priorities, or use a prioritizing process that becomes a front-end to the whole planning process.
Such a process should be based on The Law of Limiting Factors:
When a number of conditions are necessary to a process, its rate is controlled by the least favorable of these conditions.
Concentrate on the area that is limiting, not on the one for which there happens to be the most data. Frequently, there will be the least, or the least reliable, data for that limiting area—that's why it has become limiting.
Problem: Fad-driven planning.
Planning sessions are swept along by the latest software engineering fad, and the root problems at home are ignored. In a discussion on a software engineering forum, one of the correspondents noted,
"Of course technical reviews are a good idea, but we do embedded real-time systems using object-oriented analysis, so we don't need them."
Cause: Planners are easily swayed off course when they're not sure of their planning process or their data. When their self-esteem is low and they are under pressure for quick solutions to entrenched problems, they are suckers for any patent medicine huckster who rolls into town. If vendors are actually present in planning sessions, the temptation to placate them becomes irresistible to some. Others can't resist attacking them. Neither approach contributes much to successful planning.
Meta-Plan Action: Most importantly, keep vendors out of your own organization's planning sessions. To accomplish this purification, you need a process specified in the strategic plan by which vendor information can be gathered in advance and appropriately evaluated and summarized. The planning output sets the parameters to guide any necessary vendor negotiations. Of course, this advice flies in the face of strategic partnerships. In that case the partners must be consulted and kept informed, but you do not need to allow them to put their hand in your purse by allowing them in your own organization's strategy sessions—at least until you become a Congruent (Pattern 5) organization.
Remember what Bertrand Russell said: "All movements go too far." Don't get sucked in by appeals to your vanity, such as, "You can be the first, the world leader." Use your own organization's culture and business needs as the key to what you can and should do. For instance, if you haven't reached the point where you can keep track of what each project costs and how long it takes, don't even consider using some fancy estimating software. "Guess-in" always produces "fantasy-out."
1.3 Mechanics
I won't discuss the mechanics of running strategic planning sessions here, but that's not because the mechanics are unimportant. On the contrary, they're so important they are a well-explored, well-documented subject, and you can easily obtain books on the subject. Here are a few suggestions:
1. Start with Doyle and Strauss, How to Make Meetings Work: The New Interaction Method. This book lays out the fundamentals of conducting any meeting, as well as a method that can be adapted rather well to planning meetings.
2. Spencer's Winning Through Participation documents a carefully developed general method for conducting planning meetings. This method was created for use in international situations, and has since been adopted for use in much less demanding circumstances.
3. Peña's Problem Seeking: An Architectural Programming Primer. describes the problem seeking method used by these world-famous architects for initial planning of large projects. Several of my clients have adapted this method for both software engineering projects and organizational change projects.
4. Gause and Weinberg's Exploring Requirements: Quality Before Design has also been used by many of my clients as a guide to strategic planning meetings.
Each of the above approaches does a reasonable job of gathering information and creating a list of possible strategic actions, but I've watched each of them break down when it came time to negotiate priorities among the potential actions. I've come to understand that negotiating skills are in short supply among managers, and even those who possess them often fail to use them congruently. The most frequent problem is placating—saying yes to conflicting priorities or saying yes to more than the organization can possibly do.
Some books I've recommended in these situations are Laborde's Influencing with Integrity: Management Skills for Communication and Negotiation, which takes a Neurolinguistic Programming (NLP) approach, and Karass's Give and Take: The Complete Guide to Negotiating Strategies and Tactics, which takes more of an old-fashioned business approach. Finally, there is the classic from the Harvard Negotiation Project, Getting to Yes.
In the end, though, the problem with mechanics is confounding cause and effect. Organizations capable of responding rapidly and effectively tend to have a common set of values, a vision, and a strategic plan. The common misconception is that if your executives put these things into place, your organization will be successful, too. The mechanics are necessary, but they're far from sufficient, as we'll see in succeeding chapters.
1.4 Helpful Hints and Suggestions
1. Of all the temperaments, the NT Visionaries are the best planners, but they're also are the most likely to believe they can conduct successful planning meetings without information. The NF Catalysts easily fall in with their fantasies if they think that good people are involved. Yet in most software engineering organizations, the upper management and planning staff are almost all NT's, with a sprinkling of NF's. To improve your planning, search for some SJ Organizers and SP Troubleshooters to add to the process—and listen to their pleas for data.
2. (Phil Fuhrer) Technical reviews are one corrective action that applies to all of the shortcomings of strategic planning. Reviewers should be anyone who is expected to implement or track the plan, including software developers, testers, technical support staff, quality assurance people, and researchers. The plan is not done until it is reviewed by its users.
3. (Lynne Nix) If strategic plans are not to become fileware, they must eventually be translated into tactical plans that lead to action. If the tactical plans don't fit with the strategic plans, however, the strategic plans are meaningless. One way to maintain this alignment is to establish a culture of including "parent" documents with every action plan. This is a simple way to create a linked list that can be used to trace any action back to the strategic plan, or to discover that the two are unrelated.
4. (Michael Dedolph) Although we need strategies based on data, not opinion, data about opinions usually need to be taken into account. Opinions can suggest reality, and group or individual opinions can be self-fulfilling. For example, in risk assessment, it's often useful to work on the basis of what people think is the highest risk because that's the risk that will influence their actions the most. Many organizational assessment techniques measure opinions, supported to varying extent by other mechanisms. Although useful, opinion surveys have a potential drawback in that managers with extensive marketing backgrounds may try to change the opinions rather than address the sources of the problems. Advertising may induce people to try a new product, but it will rarely get people to try a counterfeit product twice.
5. Training is another general strategy that creates an adaptable organization capable of overcoming planning errors. Many organizations make the mistake of overplanning their training, becoming preoccupied with cost-effectiveness (pronounced "cost") and goal-oriented training. Although training should be applied to whatever skill goals are clearly identified, other training is needed to cover those areas that cannot be recognized from the top. An adaptable way to plan training is at a meta-level. For example, management can specify training to meet the planned needs, but in addition, each employee gets a guaranteed time and money budget for personally chosen training. Employees can pool their budgets to bring in particular persons, events, or resources. Other employees can then participate, but they have to pay their share out of their own budget. Over the year, each employee's budget has to be spent, or the employee's manager is in hefty trouble.
1.5 Summary
1. Change artists practice their skills on three levels, involving three different times scales and three different sets of skills. Change artistry "in-the-small" deals with people and problems face-to-face and day-to-day—this level is what we call tactics.
2. Tactics are not all reactions to unexpected events. Change artistry "in-the-middle" is what we call tactical planning: looking ahead to be ready to handle such events.
3. Change artistry "in-the-large" is what we call strategic planning: setting the climate in which tactical planning takes place. The strategic plan addresses two broad questions for a time interval that is typically three to five years in length:
• What products/services will we supply?
• What processes/resources/culture will we need in order to supply them?
4. The strategic planning process has to be able to estimate the feasibility of some of the visions by asking: Can we build these products with the processes/resources/culture we now have? If the answer is yes, then the process vision is one of maintaining and perfecting the present process. If the answer is no, then the planners must decide whether to reduce their ambitions or to raise their process vision.
5. When done well, the entire process of strategic planning for change can be regarded as having three major components:
• information gathering, including observing and ignoring
• problem solving, including systems thinking, negotiating, and translating into action-generating principles
• mechanics, including knowing when, where, and how the planning is done, as well as who is involved, and understanding the difference between tactics and strategy
6. The information for making strategic decisions needs to come from both inside and outside the organization. Three types of problems plague the quality of the information used in strategic planning:
• omission of some essential source
• dependence upon an unreliable source
• wallowing in excessive detail, while ignoring relevant details
7. The information for sensible change planning is simply not available until all three types of problems are cleared up. Until they are, you need to restrict the output of the plan to meta-plans: plans that need to be carried out concerning the planning process before you can do effective planning on the rest of the organization. Most of these meta-plans will concern the quality and quantity of information.
8. There are a number of typical problems of missing information:
• failing to consider customers' wishes in the planning process
• failing to consider potential customers
• inability to demonstrate any relationship between the software engineering strategy and the firm's overall business strategy.
• failing to use information of any kind from within the organization
• failing to use outside sources of information
• lack of knowledge of organizational development possibilities
• sham planning that costs a lot and disrupts the organization
8. There are a number of typical problems of using information from unreliable sources:
• using internal reports that are irrelevant or are not reliable indicators of what's happening in the organization
• measuring processes so unstable that measurements have no meaning, except to show the instability
• creating strategies based on opinion, not data
9. Even when the available information is reliable, planning sessions fall into trouble by working at the wrong level of detail:
• spending too much time on irrelevant details
• using data from different sources that are not of comparable detail
• working everything to death, while treating important issues superficially
• being swept along by the latest software engineering fad, ignoring the root problems at home
1.6 Practice
1. (Phil Fuhrer) When is a plan done? Phil's answer is, "when people no longer use it." What does he mean? Can you improve his answer?
2. (Phil Fuhrer) Planning requires information and other prerequisites. In other words, the plan needs to be planned! Write a plan for a plan including:
(1) who wants it
(2) who is affected
(3) how will the results be measured
(4) under what conditions will the plan be completed, abandoned, obsolete.
3. (Michael Dedolph) Do this at home, but only if you're willing to accept an element of risk. This exercise can have surprising and long-lasting results:
• Start by setting strategic goals for yourself, by yourself.
• Next hold a family meeting and establish a strategic plan for the family.
• Then check your own strategic goals.
• Check to see how much the family's (organizational) goals support your individual goals, and vice versa.
• Compare the levels of detail in your goals and the family plans.
This exercise may tell you something about your style as a planner. For example, did your goals change to totally reflect the family goals? Are your goals and the family's totally or partially disconnected?
If you can't complete the family planning process—if there are major spats, communication failures, or everyone's goals are totally disparate—you may want to consider engaging an organizational consultant. Now think about any parallels at work.
4. (Lynne Nix) Find your organization's latest strategic plan. (How long did that take?) Review what projects are currently underway. What is their relationship to the strategic plan? What does it take to determine this?
5. (Janice Wormington) If you've ever participated in strategic planning for your own organization, was it closer to Fig. 1-1 or 1-2? Why? How could it be improved?
6. (Michael Dedolph) How good are your listening skills? One way to tell is by having someone observe you walk around. How can they tell whether your walking around is doing harm or good? This may be an excellent (if painful) chance to assess your listening skills, but only if you can listen to your observer.
7. (Payson Hall) Managers whose work history has been exclusively with one organization tend to have difficulty being effective planners. This is because
• they are too close to see what's going on—they lack perspective.
• for better or worse, they helped get the organization where it is, and it's hard to recognize errors when you were involved in making them.
• in their career-path progression from college to programmer to analyst to senior analyst to manager, it's easy to believe that something magic happens that teaches you about working with human beings, effective interpersonal communications, negotiations, project management, leadership, and understanding your own behavior.
Brainstorm some meta-plans to deal with this problem of ingrown management.
Chapter 2. Meta-Planning: Part II. Systems Thinking
Every kind of peaceful cooperation among men is primarily based on mutual trust and only secondarily on institutions such as courts and police. - Albert Einstein
Now let's switch our metaphor from navigating to cooking; they are, after all, not so different. Superb ingredients are the first half of a recipe—necessary, but not sufficient, to create a culinary masterpiece. The second half is the skill of the chef. Without a skillful chef, all you get is a mess of fine ingredients.
In the same way, the right information at the right level using the right process, guarantees nothing about the success of change planning. As an organization grows and matures, the problems grow from baking a cupcake to creating a wedding cake. All these ingredients must be assembled with the master touch, and that is provided by clear systems thinking, perhaps a luxury when the organization is small, but an absolute necessity as it grows.
This chapter will examine some of the more common systems thinking challenges arising in strategic planning for software engineering organizations, particularly as the organization grows and matures. It will suggest the sources of these challenges, and offer some approaches available to the planning team.
2.1 Problem solving
There are many problem solving approaches ranging from individuals' styles to elaborate processes described in books or sold as high-priced proprietary systems by consulting firms. But, as they say, too many cooks spoil the broth. One approach is all you need—no more, no less. Your first challenge is deciding which one. If you can't meet that challenge, how can you imagine you could plan for an entire organization?
2.1.1 The Big Game
Problem: Who gets to tell whom what to do?
Planning sessions degenerate into arguments over whose problem solving approach to use. The key word here is "degenerate," since a reasonable amount of discussion is necessary and appropriate. You'll know a session has degenerated when it becomes unreasonable:
One of my clients delayed strategic planning for more than three years while two Senior Vice-Presidents battled over whose captive consultant would lead the sessions.
Cause: This has nothing to do with problem-solving, but with the Big Game—who gets to tell whom what to do.
Meta-Plan Action: Comment on the situation. For instance, "It seems to me that we have a conflict over whose approach to use. I believe we're lucky to have more than one approach available. If we are intelligent enough to do strategic planning, we are intelligent enough to get the best out of any of these approaches." Then stop talking and wait. Repeat as necessary.
2.1.2 No systematic approach for decision making
Problem: Inability to make decisions.
Planning sessions may move smoothly through data sharing and setting priorities, but bog down when it comes time to decide on actions.
Cause: Many planning approaches focus excessively on data gathering, and lack any systematic process for actually solving problems, or even defining them.
Meta-Plan Action: Apply a four-step approach I use for the problem-solving phase of strategic planning:
1. Define a problem in terms of a difference between what is perceived and what is desired, and when. For instance, we perceive we are spending $5.5 million a year reworking our products after they are shipped. We desire to spend no more than $1 million within three years.
2. Develop a diagram of effects relating the perceived/desired variables to other variables. In the example, this diagram would show what variables influence the cost of rework after shipping, such as the number of faults shipped, the design debt, the software engineering aptitude of the involved organizations, and the effectiveness of the process for dealing with customer problems (Figure 2-1).
Figure 2-1. A diagram of effects can be used to relate the perceived or desired variables to other variables. In this instance, the number of faults shipped and the design debt tend to increase the cost of rework after shipping. The software engineering aptitude and the effectiveness of customer support tend to decrease it (as indicated by the gray dots).
3. Examine the diagram of effects to discover the dynamic holding the perceived situation in place. If you can't discover that dynamic, then discover what is preventing you from discovering it. For instance, design debt makes rework more costly, but costly rework leads to shortcuts, which in turn lead to more design debt (Figure 2-2). Moreover, design debt leads to more errors shipped, which raises the cost of rework, which increases design debt. A third loop describes one company's practice of using the best people from product support to do the rework, thus reducing the effectiveness of customer support, thus increasing the rework. These three loops tend to raise the cost of rework, even when we try to lower it directly.
Figure 2-2. An elaborated diagram of effects can be used to discover the dynamics locking the perceived variable in place.
4. Once you understand the dynamic, identify choice points where you can break the stability control of the perceived values. These define your strategic actions. For instance, you might determine that the practice of assigning customer support people to rework is counterproductive, and may thus plan to add good people to customer support when demands for rework start to increase. Or you could develop a different plan that somehow decouples customer support from the rework cost. If you don't understand the dynamic, identify actions that will obtain the information that will allow you to discover the dynamic. In this case, these data-gathering actions become part of your strategic plan. If, for instance, you don't know why rework costs are so high, set in place a measurement team to model the influences on rework costs. For an organization that doesn't already have the appropriate measurements in place, this new measurement process is the starting point for modeling the problem in the next planning exercise.
2.2 Growth and Size
Development seems always correlated in nature with growth. That is, the shape of a thing is the result of the relationship among the growth rates of its various parts. This is in contrast to organizational changes (or change attempts) that are supposed to take place just by writing them down in a strategic plan. In nature, the elaboration of the same form to a larger size eventually doesn't work at all; an insect the size of a dog could not oxygenate its tissues, and a blade of grass the size of a redwood would merely bend over and serve as mulch on the forest floor. For similar reasons, planners often discover trouble with their planning approach when the system grows larger, or faster, and the different parts are not longer in sensible proportions.
2.2.1 Growth produces bigness
Problem: Bigger is not just the same as smaller.
As quality improves, the business improves, then grows. But growth produces bigness, which often has negative effects on quality.
Cause: Figure 2-3 is similar to Deming's chain reaction, but has two feedback loops. In software organizations, economies of scale aren't as strong as in manufacturing, so the loop through "Cost Efficiency" is weaker. On the other hand, as a software business grows, systems grow and quality becomes harder to maintain, so eventually this feedback loop becomes self-limiting.
Figure 2-3. Based on Deming's "chain reaction" but with feedback added, this effects diagram shows how quality can lead to growth, which can lead to limits on quality. As the left-hand loop (bigness can destroy quality) comes to dominate the power of the right-hand loop (economies of scale), it becomes counterproductive for the system to grow larger.
Meta-Plan Action: As Figure 2-3 indicates, the translation of market appeal into growth rate is a management choice. It's merely a cop-out to say "we must grow at the fastest possible rate." The strategic planning team must make decisions on what growth rate can be handled, and what explicit actions will be taken to control the quality level as the organization grows.
2.2.2 Complexity restricts development
Problem: Increases in complexity make a system harder to improve.
As we add explicit mechanisms to control quality, the organization seems to be harder to organize further.
Cause: Minot's Law, from biology, says that the rate of growth of an organism begins a steady decline from the moment of conception (Figure 2-4):
This suggests a general principle of organization: that once a system becomes organized, it becomes progressively more difficult to reorganize the system. That is, organization inhibits reorganization. Further, organization can be strongly modified only when active processes of organization are going on, and this accounts for critical periods of development.
Figure 2-4. Minot's Law, extended to organizational growth, says that management's efforts to raise quality by successful organization may succeed for a while, but may also produce a more complex organization, which becomes harder to organize for additional improvements. Thus, the current changes eventually become growth-rate-limiting structures for future changes.
Meta-Plan Action: In the strategic plan, avoid adding complexity that doesn't contribute to the organization's goals. Applying a bias toward simplification will allow the organization to prosper longer without feeling the limiting effects of Minot's Law.
Secondly,understand the theory of critical periods of development, which says that early small decisions about the organization may have an enormous impact on the ultimate success of the organization. So, early in the organization's life, realize that strategic planning can have great positive impact, but at the cost of great danger of negative impact. Perhaps that's why new organizations rarely do strategic planning, and most of them that try fail in their attempts.
Later on, it's much harder for organizations to change, for good or bad, so strategic planning isn't as critical. Organizations have lots of capital to tide them over mistakes, capital in the broadest sense of "knowledge imposed on the physical world." But, as Boulding said "Capital is frozen knowledge," so, capital readily becomes one of the impediments to change, although it is often thought to foster change in some magical way: "If we only had the resources." Very few successful changes can be obtained just by throwing resources at them. To succeed, organizations must acquire new and flexible knowledge to the capital they've already frozen.
2.2.3 Size restricts freedom
Problem: Overplanning stomps creativity.
People feel that because of overplanning, they are losing their opportunity to be creative.
Cause: Of course, overplanning can happen in any size organization, but beyond a certain point, freedom diminishes as numbers increase within a finite space. For example, in heavy traffic, the individual driver loses freedom of choice. To reduce traffic density, however, we may have to restrict freedom of choice to be on the road at all. Or, closer to home, as more people are updating the source code library, we have to restrict the types of usage, or else we have to restrict use to specially trained people.
Meta-Plan Action: These effects have nothing to do with planning, but with scale—unless the planning is poorly done. You must make the trade-offs, but then make them public so people will understand what they are getting for the price they are paying.
Be careful of the tendency to make changes of wider scope than they really need to be. Examine your motives: Overplanning can be a power game. If it isn't, it often looks like one to the people down below. When planning some standard or constraint, ask, "What is the narrowest scope we can use to accomplish our goal?"
Increased depth creates the same effect as increased scope, so don't succumb to the temptation to micro-plan. If, for example, you plan to the level of five-person teams, rather than the level of the individual, you reduce the effective size of the plan by a factor of five or so. If you plan to the level of fifty-person departments, you reduce the effective size by a factor of fifty. Such macro-planning requires trust in your teams and department managers, and, more important, it demonstrates trust.
2.2.4 Tools influence thought
Problem: Our planning tools have become overly complicated.
Planning projections don't seem to work as well as they used to. As one frustrated manager told me, waving a five-pound sheaf of spreadsheets in my face:
Five years ago, I was all in favor of introducing spreadsheet models into our planning. Now, however, we have a three-person department that does nothing but create seven-decimal-place spreadsheets between planning sessions. If I could put a bug in all our spreadsheet programs so they'd all stop working, I'd do it without hesitation.
Cause: Planning tools may be the problem, because they don't scale up as the system size grows. Spreadsheets, for example, are most easily used as linear projection tools, and usually produce bogus results when the system goes nonlinear. And, of course, growth is always nonlinear.
More sophisticated modeling tools use more complex models, but because of this complexity, they tend to become black boxes to their users.
Meta-Plan Action: Examine your planning tools. View spreadsheet projects with particular suspicion. Open your modeling tools and study the assumptions upon which they were based. Update them, or discard them.
2.2.5 Big isn't the same as small
Problem: Growth produces bigness.
To illustrate one problem of growing organizations, here's a quote taken from an e-mail message from a potential client:
I can't figure out what's happening around here. We've been a successful small company, but now nothing seems to be working right. Meetings take longer, but critical people are often absent so that parts of the meeting have to be repeated. People argue more for their own area of responsibility and don't listen to each other. Key data can't be found, or everybody has their own incompatible version. Suboptimization seems to be the default solution to everything.
Cause: Centuries ago, Galileo described The Principle of Similitude:
• With increased size, the ratio of surface to volume decreases.
• Different growing shapes will have different balance points of surface and volumetric effects.
Galileo's Principle of Similitude applies well to the growing organization. "Volume" is the internal part of the organization. "Surface" is the interface between the organization and its outside. As the organization grows, its relationship with the outside is strained as it tries to maintain its internal viability.
This principle applies to the entire organization and its relationships to customers and vendors, but it also applies to the internal parts of the organization. Communication among parts becomes strained, and each part tends to become a feudal domain that seems surrounded by high, thick walls, or even moats. It seems that way, but the domain is probably just concentrating on its internal problems.
Sometimes, creating and protecting a feudal group is an effective strategy, as when creating islands of excellence, or at least islands of competence, that will eventually outlive the larger organization. More often, though, upper management is unconsciously encouraging this trend by dispensing rewards to individual groups with no coherent strategy and no attention to the systemic ill-effects.
Meta-Plan Action: Within the organization, keep groups as small as possible, and plan activities that regularly cross groups, such as requiring one outsider in each technical or project review. Communication with the outside must be given as a sole responsibility to people trained for the job and measured by their communication success.
Do not attempt to counter the Principle of Similitude with posters, slogans, and pep talks. The forces of nature will always overcome words, and the words will backfire and produce contempt and ridicule for management. Set rewards based on organizational performance, not on the performance of individual groups.
2.3 Risk and Reward
Strategic
planners need to think in terms of risk and reward trade-offs. One of
the most common trade-off decisions in strategic planning is between
added value and probability of success (Figure 2-5). By taking a
chance on a new technology, you may get a great increase in value or
a decrease in costs. But new technologies are risky and, if you fail,
your payoff may be nothing at all, or actually negative.
Figure 2-5. In choosing a way to go about engineering a certain product, first you need to consider the trade-off between risk and reward. Points on the curve represent the combination of risks and rewards for the best ways of doing things. We may not do this well, but we can't do any better. (Point X is just one point in the area above the curve that's impossible to reach. A plan that specifies point X is an impossible plan.) It is up to the management to choose the point on which it wishes to operate—A or B or somewhere else—and then to plan and act consistently with that choice.
Still, the choice of position on the curve of Figure 2-5 depends on the organization's situation, which is a good example of why your planners need to know the organization's financial position, the engineering competence level, and how both stand with respect to the rest of the industry.
2.3.1 Always be first
Problem: Our survival depends on our speed.
A start-up software company has much to lose if its leaders don't do something spectacular, and fast.
Cause: They may need to take a high risk in order to get a high enough reward to stay in existence. Even if the organization doesn't need such a high reward, the owners of the company may not be interested in simply surviving in a small niche, with a product that's just a little different from everyone else's.
Meta-Plan Action: Managers will take actions that establish a gospel of high risk, high reward, and those actions, if effective, will create the culture that fits these goals. For instance, they may plan to
• pay employees with low salaries, but high stock options
• set ambitious schedules to beat competitors to the market (but if the schedules are unreasonable, they guarantee failure to deliver)
• promise outrageous levels of functionality to dominate every competitor (but if the levels are so outrageous that they can't deliver, they disappoint everybody, and perhaps become a laughing stock)
• approve only the newest, most innovative tools and processes, even though they are untried (which increases their chance of failing because of tool failure).
The problem with such a strategy is to keep it aggressive but reasonable. Anyone can sell anything, as long as they don't have to deliver it.
2.3.2 Always be second
Problem: We have more to lose than to gain.
The high-level managers of a large, established software provider have much to lose if they make high risk plans.
Cause: Someone once said that IBM's secret motto was, "Always be second." At least, that's the way these planners look to the outside. But to the inside, they make perfect sense: Don't stick your neck out, but once someone else has taken the big risk and succeeded, be ready to move in a swift, sure-footed way.
Meta-Plan Action: A planning group in a large established company might create a contrasting set of cultural decisions:
• Pay competitive starting salaries with great benefits, but offer no stock options.
• Set modest schedules, but block competitors with elaborate pre-announcements (if you don't mind being unethical).
• Dominate competitors with vague sales promises of "We know what's best."
• Approve only those tools and processes that have stood the test of time.
One side-effect of this strategy: Good people may find it stifling or ethically unacceptable. Over a long period of time—extended by the large pool of capital—the organization's advantage may erode, and only be noticed when it's too late to do anything about it. Thus, one additional strategy should be used:
• Invest a small but protected percentage of your income in activities that explore—in a safe way—the newest and the most spectacular. Various forms for this investment include an Advanced Technology Group, a Software Engineering Process Group, or a Research Group. At the very least, this strategy will give you a group of people who are ready to move quickly when one of the new approaches proves sound.