FLOOT Lifecycle & Techniques

The Full-Lifecycle Object-Oriented Testing (FLOOT) methodology is a collection of testing techniques to verify and validate object-oriented software. The FLOOT lifecycle is depicted in Figure 1, indicating a wide variety of techniques (described in Table) are available to you throughout all aspects of software development.   The list of techniques is not meant to be complete – instead the goal is to make it explicit that you have a wide range of options available to you. It is important to understand that although the FLOOT method is presented as a collection of serial phases it does not need to be so: the techniques of FLOOT can be applied with evolutionary/agile processes as well.  The reason why I present the FLOOT in a “traditional” manner is to make it explicit that you can in fact test throughout all aspects of software development, not just during coding.

Table 1. Testing techniques.

FLOOT Technique	Description
Black-box testing	Testing that verifies the item being tested when given the appropriate input provides the expected results.
Boundary-value testing	Testing of unusual or extreme situations that an item should be able to handle.
Class testing	The act of ensuring that a class and its instances (objects) perform as defined.
Class-integration testing	The act of ensuring that the classes, and their instances, form some software perform as defined.
Code review	A form of technical review in which the deliverable being reviewed is source code.
Component testing	The act of validating that a component works as defined.
Coverage testing	The act of ensuring that every line of code is exercised at least once.
Design review	A technical review in which a design model is inspected.
Inheritance-regression testing	The act of running the test cases of the super classes, both direct and indirect, on a given subclass.
Integration testing	Testing to verify several portions of software work together.
Method testing	Testing to verify a method (member function) performs as defined.
Model review	An inspection, ranging anywhere from a formal technical review to an informal walkthrough, by others who were not directly involved with the development of the model.
Path testing	The act of ensuring that all logic paths within your code are exercised at least once.
Prototype review	A process by which your users work through a collection of use cases, using a prototype as if it was the real system. The main goal is to test whether the design of the prototype meets their needs.
Prove it with code	The best way to determine if a model actually reflects what is needed, or what should be built, is to actually build software based on that model that show that the model works.
Regression testing	The acts of ensuring that previously tested behaviors still work as expected after changes have been made to an application.
Stress testing	The act of ensuring that the system performs as expected under high volumes of transactions, users, load, and so on.
Technical review	A quality assurance technique in which the design of your application is examined critically by a group of your peers. A review typically focuses on accuracy, quality, usability, and completeness. This process is often referred to as a walkthrough, an inspection, or a peer review.
Usage scenario testing	A testing technique in which one or more person(s) validate a model by acting through the logic of usage scenarios.
User interface testing	The testing of the user interface (UI) to ensure that it follows accepted UI standards and meets the requirements defined for it. Often referred to as graphical user interface (GUI) testing.
White-box testing	Testing to verify that specific lines of code work as defined. Also referred to as clear-box testing.

I’d like to share a few philosophies with regards to testing:

1. The goal is to find defects.  The primary purpose of testing is to validate the correctness of whatever it is that you’re testing.  In other words, successful tests find bugs.
2. You can validate all artifacts.  As you will see in this chapter, you can test all your artifacts, not just your source code. At a minimum you can review models and documents and therefore find and fix defects long before they get into your code.
3. Test often and early.  The potential for the cost of change to rise exponentially motivates you to test as early as possible.
4. Testing builds confidence.  In Extreme Programming Explained Kent Beck makes an interesting observation that when you have a full test suite, a test suite is a collection of tests, and if you run it as often as possible, that it gives you the courage to move forward.  Many people fear making a change to their code because they’re afraid that they’ll break it, but with a full test suite in place if you do break something you know you’ll detect it and then fix it.
5. Test to the risk of the artifact.  The riskier something is, the more it needs to be reviewed and tested. In other words you should invest significant effort testing in an air traffic control system but nowhere near as much effort testing a “Hello World” application.

One test is worth a thousand opinions.  You can tell me that your application works, but until you show me the test results, I will not believe you.

Have you tested your Test strategy lately?

Ten timeless tests can help you kick the tires on your strategy, and kick up the level of strategic dialogue throughout your company.

“What’s the next new thing in strategy?” a senior executive recently asked Phil Rosenzweig, a professor at IMD,¹ in Switzerland. His response was surprising for someone whose career is devoted to advancing the state of the art of strategy: “With all respect, I think that’s the wrong question. There’s always new stuff out there, and most of it’s not very good. Rather than looking for the next musing, it’s probably better to be thorough about what we know is true and make sure we do that well.”

Let’s face it: the basic principles that make for good strategy often get obscured. Sometimes the explanation is a quest for the next new thing—natural in a field that emerged through the steady accumulation of frameworks promising to unlock the secret of competitive advantage.² In other cases, the culprit is torrents of data, reams of analysis, and piles of documents that can be more distracting than enlightening.

Ultimately, strategy is a way of thinking, not a procedural exercise or a set of frameworks. To stimulate that thinking and the dialogue that goes along with it, we developed a set of tests aimed at helping executives assess the strength of their strategies. We focused on testing the strategy itself (in other words, the output of the strategy-development process), rather than the frameworks, tools, and approaches that generate strategies, for two reasons. First, companies develop strategy in many different ways, often idiosyncratic to their organizations, people, and markets. Second, many strategies emerge over time rather than from a process of deliberate formulation.³

There are ten tests on our list, and not all are created equal. The first—“will it beat the market?”—is comprehensive. The remaining nine disaggregate the picture of a market-beating strategy, though it’s certainly possible for a strategy to succeed without “passing” all nine of them. This list may sound more complicated than the three Cs or the five forces of strategy.⁴ But detailed pressure testing, in our experience, helps pinpoint more precisely where the strategy needs work, while generating a deeper and more fruitful strategic dialogue.

Those conversations matter, but they often are loose and disjointed. We heard that, loud and clear, over the past two years in workshops where we explored our tests with more than 700 senior strategists around the world. Furthermore, a recent McKinsey Quarterly survey of 2,135 executives indicates that few strategies pass more than three of the tests (Exhibit 1). In contrast, the reflections of a range of current and former strategy practitioners (see “How we do it: Strategic tests from four senior executives”) suggest that the tests described here help formalize something that the best strategists do quite intuitively.

The tests of a good strategy are timeless in nature. But the ability to pressure-test a strategy is especially timely now. The financial crisis of 2008 and the recession that followed made some strategies obsolete, revealed weaknesses in others, and forced many companies to confront choices and trade-offs they put off in boom years. At the same time, a shift toward shorter planning cycles and decentralized strategic decision making are increasing the utility of a common set of tests.⁵ All this makes today an ideal time to kick the tires on your strategy.

Test 1: Will your strategy beat the market?

All companies operate in markets surrounded by customers, suppliers, competitors, substitutes, and potential entrants, all seeking to advance their own positions. That process, unimpeded, inexorably drives economic surplus—the gap between the return a company earns and its cost of capital—toward zero.

For a company to beat the market by capturing and retaining an economic surplus, there must be an imperfection that stops or at least slows the working of the market. An imperfection controlled by a company is a competitive advantage. These are by definition scarce and fleeting because markets drive reversion to mean performance (Exhibit 2). The best companies are emulated by those in the middle of the pack, and the worst exit or undergo significant reform. As each player responds to and learns from the actions of others, best practice becomes commonplace rather than a market-beating strategy. Good strategies emphasize difference—versus your direct competitors, versus potential substitutes, and versus potential entrants.

 

Market participants play out the drama of competition on a stage beset by randomness. Because the evolution of markets is path dependent—that is, its current state at any one time is the sum product of all previous events, including a great many random ones—the winners of today are often the accidents of history. Consider the development of the US tire industry. At its peak in the mid-1920s, a frenzy of entry had created almost 300 competitors. Yet by the 1940s, four producers controlled more than 70 percent of the market. Those winners happened to make retrospectively lucky choices about location and technology, but at the time it was difficult to tell which companies were truly fit for the evolving environment. The histories of many other industries, from aerospace to information technology, show remarkably similar patterns.

To beat the market, therefore, advantages have to be robust and responsive in the face of onrushing market forces. Few companies, in our experience, ask themselves if they are beating the market—the pressures of “just playing along” seem intense enough. But playing along can feel safer than it is. Weaker contenders win surprisingly often in war when they deploy a divergent strategy, and the same is true in business.

Test 2: Does your strategy tap a true source of advantage?

Know your competitive advantage, and you’ve answered the question of why you make money (and vice versa). Competitive advantage stems from two sources of scarcity: positional advantages and special capabilities.

Positional advantages are rooted in structurally attractive markets. By definition, such advantages favor incumbents: they create an asymmetry between those inside and those outside high walls. For example, in Australia, two beer makers control 95 percent of the market and enjoy triple the margins of US brewers. This situation has sustained itself for two decades, but it wasn’t always so. Beginning in the 1980s, the Australian industry experienced consolidation. That change in structure was associated with a change in industry conduct (price growth began outstripping general inflation) and a change in industry performance (higher profitability). Understanding the relationship among structure, conduct, and performance is a critical part of the quest for positional advantage.

Special capabilities, the second source of competitive advantage, are scarce resources whose possession confers unique benefits. The most obvious resources, such as drug patents or leases on mineral deposits, we call “privileged, tradable assets”: they can be bought and sold. A second category of special capabilities, “distinctive competencies,” consists of things a company does particularly well, such as innovating or managing stakeholders. These capabilities can be just as powerful in creating advantage but cannot be easily traded.

Too often, companies are cavalier about claiming special capabilities. Such a capability must be critical to a company’s profits and exist in abundance within it while being scarce outside. As such, special capabilities tend to be specific in nature and few in number. Companies often err here by mistaking size for scale advantage or overestimating their ability to leverage capabilities across markets. They infer special capabilities from observed performance, often without considering other explanations (such as luck or positional advantage). Companies should test any claimed capability advantage vigorously before pinning their hopes on it.

When companies bundle together activities that collectively create advantage, it becomes more difficult for competitors to identify and replicate its exact source. Consider Aldi, the highly successful discount grocery retailer. To deliver its value proposition of lower prices, Aldi has completely redesigned the typical business system of a supermarket: only 1,500 or so products rather than 30,000, the stocking of one own-brand or private label rather than hundreds of national brands, and superlean replenishment on pallets and trolleys, thus avoiding the expensive task of hand stacking shelves. Given the enormous changes necessary for any supermarket that wishes to copy the total system, it is extremely difficult to mimic Aldi’s value proposition.

Finally, don’t forget to take a dynamic view. What can erode positional advantage? Which special capabilities are becoming vulnerable? There is every reason to believe that competitors will exploit points of vulnerability. Assume, like Lewis Carroll’s Red Queen, that you have to run just to stay in the same place.

Test 3: Is your strategy granular about where to compete?

The need to beat the market begs the question of which market. Research shows that the unit of analysis used in determining strategy (essentially, the degree to which a market is segmented) significantly influences resource allocation and thus the likelihood of success: dividing the same businesses in different ways leads to strikingly different capital allocations.

What is the right level of granularity? Push within reason for the finest possible objective segmentation of the market: think 30 to 50 segments rather than the more typical 5 or so. Too often, by contrast, the business unit as defined by the organizational chart becomes the default for defining markets, reducing from the start the potential scope of strategic thinking.

Defining and understanding these segments correctly is one of the most practical things a company can do to improve its strategy. Management at one large bank attributed fast growth and share gains to measurably superior customer perceptions and satisfaction. Examining the bank’s markets at a more granular level suggested that 90 percent of its outperformance could be attributed to a relatively high exposure to one fast-growing city and to a presence in a fast-growing product segment. This insight helped the bank avoid building its strategy on false assumptions about what was and wasn’t working for the operation as a whole.

In fact, 80 percent of the variance in revenue growth is explained by choices about where to compete, according to research summarized in The Granularity of Growth, leaving only 20 percent explained by choices about how to compete. Unfortunately, this is the exact opposite of the allocation of time and effort in a typical strategy-development process. Companies should be shifting their attention greatly toward the “where” and should strive to outposition competitors by regularly reallocating resources as opportunities shift within and between segments.

Test 4: Does your strategy put you ahead of trends?

The emergence of new trends is the norm. But many strategies place too much weight on the continuation of the status quo because they extrapolate from the past three to five years, a time frame too brief to capture the true violence of market forces.

A major innovation or an external shock in regulation, demand, or technology, for example, can drive a rapid, full-scale industry transition. But most trends emerge fairly slowly—so slowly that companies generally fail to respond until a trend hits profits. At this point, it is too late to mount a strategically effective response, let alone shape the change to your advantage. Managers typically delay action, held back by sunk costs, an unwillingness to cannibalize a legacy business, or an attachment to yesterday’s formula for success. The cost of delay is steep: consider the plight of major travel agency chains slow to understand the power of online intermediaries. Conversely, for companies that get ahead of the curve, major market transitions are an opportunity to rethink their commitments in areas ranging from technology to distribution and to tailor their strategies to the new environment.

To do so, strategists must take trend analysis seriously. Always look to the edges. How are early adopters and that small cadre of consumers who seem to be ahead of the curve acting? What are small, innovative entrants doing? What technologies under development could change the game? To see which trends really matter, assess their potential impact on the financial position of your company and articulate the decisions you would make differently if that outcome were certain. For example, don’t just stop at an aging population as a trend—work it through to its conclusion. Which consumer behaviors would change? Which particular product lines would be affected? What would be the precise effect on the P&L? And how does that picture line up with today’s investment priorities?

Test 5: Does your strategy rest on privileged insights?

Data today can be cheap, accessible, and easily assembled into detailed analyses that leave executives with the comfortable feeling of possessing an informed strategy. But much of this is noise and most of it is widely available to rivals. Furthermore, routinely analyzing readily available data diverts attention from where insight-creating advantage lies: in the weak signals buried in the noise.

In the 1990s, when the ability to burn music onto CDs emerged, no one knew how digitization would play out; MP3s, peer-to-peer file sharing, and streaming Web-based media were not on the horizon. But one corporation with a large record label recognized more rapidly than others that the practical advantage of copyright protection could quickly become diluted if consumers began copying material. Early recognition of that possibility allowed the CEO to sell the business at a multiple based on everyone else’s assumption that the status quo was unthreatened.

	The insight deficit

Developing proprietary insights isn’t easy. In fact, this is the element of good strategy where most companies stumble (see sidebar, “The insight deficit”). A search for problems can help you get started. Create a short list of questions whose answers would have major implications for the company’s strategy—for example, “What will we regret doing if the development of India hiccups or stalls, and what we will not regret?” In doing so, don’t forget to examine the assumptions, explicit and implicit, behind an established business model. Do they still fit the current environment?

Another key is to collect new data through field observations or research rather than to recycle the same industry reports everyone else uses. Similarly, seeking novel ways to analyze the data can generate powerful new insights. For example, one supermarket chain we know recently rethought its store network strategy on the basis of surprising results from a new clustering algorithm.

Finally, many strategic breakthroughs have their root in a simple but profound customer insight (usually solving an old problem for the customer in a new way). In our experience, companies that go out of their way to experience the world from the customer’s perspective routinely develop better strategies.

Test 6: Does your strategy embrace uncertainty?

A central challenge of strategy is that we have to make choices now, but the payoffs occur in a future environment we cannot fully know or control. A critical step in embracing uncertainty is to try to characterize exactly what variety of it you face—a surprisingly rare activity at many companies. Our work over the years has emphasized four levels of uncertainty. Level one offers a reasonably clear view of the future: a range of outcomes tight enough to support a firm decision. At level two, there are a number of identifiable outcomes for which a company should prepare. At level three, the possible outcomes are represented not by a set of points but by a range that can be understood as a probability distribution. Level four features total ambiguity, where even the distribution of outcomes is unknown.

In our experience, companies oscillate between assuming, simplistically, that they are operating at level one (and making bold but unjustified point forecasts) and succumbing to an unnecessarily pessimistic level-four paralysis. In each case, careful analysis of the situation usually redistributes the variables into the middle ground of levels two and three.

Rigorously understanding the uncertainty you face starts with listing the variables that would influence a strategic decision and prioritizing them according to their impact. Focus early analysis on removing as much uncertainty as you can—by, for example, ruling out impossible outcomes and using the underlying economics at work to highlight outcomes that are either mutually reinforcing or unlikely because they would undermine one another in the market. Then apply tools such as scenario analysis to the remaining, irreducible uncertainty, which should be at the heart of your strategy.

Test 7: Does your strategy balance commitment and flexibility?

Commitment and flexibility exist in inverse proportion to each other: the greater the commitment you make, the less flexibility remains. This tension is one of the core challenges of strategy. Indeed, strategy can be expressed as making the right trade-offs over time between commitment and flexibility.

Making such trade-offs effectively requires an understanding of which decisions involve commitment. Inside any large company, hundreds of people make thousands of decisions each year. Only a few are strategic: those that involve commitment through hard-to-reverse investments in long-lasting, company-specific assets. Commitment is the only path to sustainable competitive advantage.

In a world of uncertainty, strategy is about not just where and how to compete but also when. Committing too early can be a leap in the dark. Being too late is also dangerous, either because opportunities are perishable or rivals can seize advantage while your company stands on the sidelines. Flexibility is the essential ingredient that allows companies to make commitments when the risk/return trade-off seems most advantageous.

A market-beating strategy will focus on just a few crucial, high-commitment choices to be made now, while leaving flexibility for other such choices to be made over time. In practice, this approach means building your strategy as a portfolio comprising three things: big bets, or committed positions aimed at gaining significant competitive advantage; no-regrets moves, which will pay off whatever happens; and real options, or actions that involve relatively low costs now but can be elevated to a higher level of commitment as changing conditions warrant. You can build underpriced options into a strategy by, for example, modularizing major capital projects or maintaining the flexibility to switch between different inputs.

Test 8: Is your strategy contaminated by bias?

It’s possible to believe honestly that you have a market-beating strategy when, in fact, you don’t. Sometimes, that’s because forces beyond your control change. But in other cases, the cause is unintentional fuzzy thinking.

Behavioral economists have identified many characteristics of the brain that are often strengths in our broader, personal environment but that can work against us in the world of business decision making. The worst offenders include overoptimism (our tendency to hope for the best and believe too much in our own forecasts and abilities), anchoring (tying our valuation of something to an arbitrary reference point), loss aversion (putting too much emphasis on avoiding downsides and so eschewing risks worth taking), the confirmation bias (overweighting information that validates our opinions), herding (taking comfort in following the crowd), and the champion bias (assigning to an idea merit that’s based on the person proposing it).

Strategy is especially prone to faulty logic because it relies on extrapolating ways to win in the future from a complex set of factors observed today. This is fertile ground for two big inference problems: attribution error (succumbing to the “halo effect”) and survivorship bias (ignoring the “graveyard of silent failures”). Attribution error is the false attribution of success to observed factors; it is strategy by hindsight and assumes that replicating the actions of another company will lead to similar results. Survivorship bias refers to an analysis based on a surviving population, without consideration of those who did not live to tell their tale: this approach skews our view of what caused success and presents no insights into what might cause failure—were the survivors just luckier? Case studies have their place, but hindsight is in reality not 20/20. There are too many unseen factors.

Developing multiple hypotheses and potential solutions to choose among is one way to “de-bias” decision making. Too often, the typical drill is to develop a promising hypothesis and put a lot of effort into building a fact base to validate it. In contrast, it is critical to bring fresh eyes to the issues and to maintain a culture of challenge, in which the obligation to dissent is fostered.

The decision-making process can also be de-biased by, for example, specifying objective decision criteria in advance and examining the possibility of being wrong. Techniques such as the “premortem assessment” (imagining yourself in a future where your decision turns out to have been mistaken and identifying why that might have been so) can also be useful.

Test 9: Is there conviction to act on your strategy?

This test and the one that follows aren’t strictly about the strategy itself but about the investment you’ve made in implementing it—a distinction that in our experience quickly becomes meaningless because the two, inevitably, become intertwined. Many good strategies fall short in implementation because of an absence of conviction in the organization, particularly among the top team, where just one or two nonbelievers can strangle strategic change at birth.

Where a change of strategy is needed, that is usually because changes in the external environment have rendered obsolete the assumptions underlying a company’s earlier strategy. To move ahead with implementation, you need a process that openly questions the old assumptions and allows managers to develop a new set of beliefs in tune with the new situation. This goal is not likely to be achieved just via lengthy reports and presentations. Nor will the social processes required to absorb new beliefs—group formation, building shared meaning, exposing and reconciling differences, aligning and accepting accountability—occur in formal meetings.

CEOs and boards should not be fooled by the warm glow they feel after a nice presentation by management. They must make sure that the whole team actually shares the new beliefs that support the strategy. This requirement means taking decision makers on a journey of discovery by creating experiences that will help them viscerally grasp mismatches that may exist between what the new strategy requires and the actions and behavior that have brought them success for many years. For example, visit plants and customers or tour a country your company plans to enter, so that the leadership team can personally meet crucial stakeholders. Mock-ups, video clips, and virtual experiences also can help.

The result of such an effort should be a support base of influencers who feel connected to the strategy and may even become evangelists for it. Because strategy often emanates from the top, and CEOs are accustomed to being heeded, this commonsense step often gets overlooked, to the great detriment of the strategy.

Test 10: Have you translated your strategy into an action plan?

In implementing any new strategy, it’s imperative to define clearly what you are moving from and where you are moving to with respect to your company’s business model, organization, and capabilities. Develop a detailed view of the shifts required to make the move, and ensure that processes and mechanisms, for which individual executives must be accountable, are in place to effect the changes. Quite simply, this is an action plan. Everyone needs to know what to do. Be sure that each major “from–to shift” is matched with the energy to make it happen. And since the totality of the change often represents a major organizational transformation, make sure you and your senior team are drawing on the large body of research and experience offering solid advice on change management—a topic beyond the scope of this article!

Finally, don’t forget to make sure your ongoing resource allocation processes are aligned with your strategy. If you want to know what it actually is, look where the best people and the most generous budgets are—and be prepared to change these things significantly. Effort spent aligning the budget with the strategy will pay off many times over.

As we’ve discussed the tests with hundreds of senior executives at many of the world’s largest companies, we’ve come away convinced that a lot of these topics are part of the strategic dialogue in organizations. But we’ve also heard time and again that discussion of such issues is often, as one executive in Japan recently told us, “random, simultaneous, and extremely confusing.” Our hope is that the tests will prove a simple and effective antidote: a means of quickly identifying gaps in executives’ strategic thinking, opening their minds toward new ways of using strategy to create value, and improving the quality of the strategy-development process itself.

Notes

¹ International Institute for Management Development.

² For a rich account of strategy’s birth and growth as a field, see Walter Kiechel, The Lords of Strategy, Boston, MA: Harvard Business School Press, 2010.

³ For a classic statement of the idea that strategies are more emergent than planned, see Henry Mintzberg, “Crafting strategy,” Harvard Business Review, 1987, July–August, Volume 65, Number 4, pp. 66–75.

⁴ The three Cs and the five forces are seminal strategy frameworks. The three Cs (competitors, customers, and company) were articulated by retired McKinsey partner Kenichi Ohmae in The Mind of the Strategist (McGraw-Hill, 1982). The five forces (barriers to entry, buyer power, supplier power, the threat of substitutes, and the degree of rivalry) were set forth by Harvard Business School professor Michael Porter in Competitive Strategy(Free Press, 1998).