Scrumerati

Scrum

Abstract

Scrum is an agile project management technique with high transparency, adaptive control, reasonably accurate release forecasting, and high productivity. It helps organize efforts involving teams and creativity (including software engineering, product design, venture capital management, charities, civic organizations, marketing communications, etc.). Because Scrum exposes managers and team members to greater visibility and accountability than traditional waterfall approaches, and because it requires different management structures, some organizations encounter resistance implementing it. Scrum focuses on management, so it can be combined with field-specific agile techniques, such as Extreme Programming.

Problem

The most common project management technique today is called "waterfall", reflecting that development flows from phase to phase, ending in the release of a completed project. In waterfall management, all the requirements are gathered up-front. Requirements gathering can take months. Product managers consider the existing marketplace, hypothesize future needs when the product will be released, estimate the development time needed to implement each requirement and analyze dependencies between development activities, putting together a Gantt chart that forecasts a release date. The development team develops the product, then the QA department tests the product, then the system is released as a beta, then bug reports and feature requests come in, then developers change the system, and out pops a completed product. Right on time. In theory.

However, once development starts, the Gantt chart typically loses credibility rapidly. Developers working under optimistic assumptions discover implementation difficulties—the creative process is not so easy to predict. Product requirements thought to be important at project initiation become irrelevant, superseded by competitive forces, new market developments, or better market understanding. The product managers add new requirements to the list. Finally, when the customers view the beta, surprises ensue. Typically, customers provide new requirements no one imagined, and ignore features all assumed were essential. The release goes out late. In an ongoing effort to "fix the situation" with the next product release, management often demands more meetings and more advance planning, taking more time before development starts.

Historically, waterfall forecasts are notoriously wrong; no one is shocked by projects forecast to release in 18 months that took 36 months. With late releases, organizations often sacrifice quality for speed. Companies can achieve bankruptcy this way; developers can lose their jobs and careers. In some situations, people have likely died.[6]

Solution

To solve this problem, reconsider some assumptions. If requirements obtained upfront are likely to radically change, a system that adapts rapidly to change would likely work better. If developer work estimates are reliably bad, find a way to map developer work estimates into actual developer time. If expensive advance planning results in marginally better or worse results than no planning at all, then plan over shorter horizons rather than longer.

Scrum is a software management and planning technique that balances these goals:

• Stakeholders can re-prioritize developer efforts at least monthly,
• Outsiders can experiment with full implementations of some features first hand at least monthly,
• Developers can measure their productivity empirically,
• Stakeholders can forecast release dates based on feature set, or tune the feature set to meet a release date,
• Developers can focus on their work uninterrupted, while simultaneously communicating beneficially in highly controlled, structured meetings, and
• Developers can refine their processes and work activities through feedback loops and self-organization.

Scrum was originally developed by Jeff Sutherland [1] and Ken Schwaber[2]

People

A Scrum Development Team comprises a Product Owner, between three and seven Developers, and a ScrumMaster. The Team is jointly responsible for the product, not only functionality, but quality, releasability, documentation, etc.

Team members are called Pigs; outsiders are called Chickens. The terminology arises from a joke: A chicken proposes to a pig that they form a restaurant together called "Ham and Eggs." The pig demurs, stating "I don't think it's a good idea. You'd just be involved, but I'd be committed."

The Product Owner in Scrum articulates stakeholder needs in the form of Product Backlog Items or User Stories, and assembles those Product Backlog Items into a prioritized list called a Product Backlog. There is only one Product Owner per Team (but several teams can have a single Product Owner), ideally there is only one primary Product Owner per Product (which may involve multiple Teams). Having only one Product Owner provides a "single wringable neck" for upper management, and a single authoritative source of prioritization and clarification for the Team. The traditional equivalent of the Product Owner is the "Product Manager."

Developers in Scrum include software engineers, user-interface designers, quality assurance engineers, database engineers, operations "shepherds", and other people who commit to a significant amount of work for the team. Ideally, development efforts are self-contained within the Team, so outsiders are not required to produce a releasable product. Furthermore, Teams and Developers are encouraged to be self-contained and cross-functional. If Developers don't know a skill, but that skill is needed by the team, they are encouraged to spend the time to learn and apply it.

The ScrumMaster facilities self-organization, supports the Team and Product Owner, enforces the process, encourages activities that improve the quality and likelihood of developing a releasable product, and serves as a sacrificial lamb to interruptions that would otherwise defocus developers. The ScrumMaster protects Developers from direct appeals from the Product Owner or other stakeholders. This supports Developer productivity: when Developers are interrupted, their productivity can drop dramatically. The ScrumMaster also works on behalf of developers to remove impediments, especially when those impediments arise from outside the team. The ScrumMaster is responsible for pro-actively providing transparency into the Team's progress and historic productivity information to outside stakeholders. The ScrumMaster encourages the team to improve its practices and make its work more feasibly releasable.

Process

Developers attempt to produce a Feasibly Releasable Product in a fixed time-period, called a Sprint. Each Sprint contains a Sprint Planning Meeting, Sprint Work, a Sprint Review (or Sprint Demo) and a Sprint Retrospective. Sprints occur in succession; in most cases there is no delay between the end of a Sprint and the beginning of the next.

Sprint Cycle

>Using the same duration for successive Sprints provides for easier planning and more reliable release forecasting. The maximum Sprint duration is a month; other common choices are four-weeks, three-weeks and two-weeks. In my experience, new practitioners of Scrum often start with longer Sprints. As they gain facility with Scrum and incorporate more efficient quality automation, they can reduce the duration.

All Scrum meetings are Time Boxed, meaning that they start exactly on time, and they last no more than a pre-established duration. A good meeting leader (in Scrum, almost always the ScrumMaster) monitors a meeting's quality: when its cost begins exceeding its value or veers from its goal, the leader should immediately discuss that observation with the team, and help redirect conversation or end the meeting. Meeting leaders may jot down "topics for later" to keep a meeting on track, inviting optional ad hoc discussion on those topics after the official meeting ends. If a meeting leader is not present, another member starts and runs the meeting. If members arrive late, they are penalized (a fine donated to charity is common). By enforcing start times vigorously, the process recognizes the enormous cost of idle developers, not only in squandered salary but also in defocused productivity.

Sprint Planning Meeting

In the first half of a Sprint Planning Meeting, the Product Owner presents the highest priority Product Backlog Items. For each item, the Product Owner describes it to Team member satisfaction, and obtains a jointly established effort estimate from the team. Many teams use Planning Poker to obtain this estimate[3]. New practitioners typically estimate Product Backlog Item work effort in units of Ideal Developer Days, but sophisticates use Story Points. When more than a Sprint's worth of effort has been estimated in total, the team selects a subset of high-priority backlog items it believes it can complete in the Sprint. This subset is called the Sprint Backlog; it is comprised of Sprint Backlog Items. With the Product Owner's consent, the Team can choose to skip a high-priority item. This can help focus the Sprint on a single theme, or defer items that need further evaluation.

As an Arab or Israeli user, I want the system to display text right-to-left.

1. Code unit-tests and right-to-left language support in client application
2. Create Arabic and Hebrew localization strings
3. Store Arabic and Hebrew strings in properties files
4. Code unit-tests first, then code web server transmitting localization strings to client
   
5. Design acceptable user-experience for Arabic and Hebrew language and users
6. Code unit-tests first, then code web server providing designed user-experience
   
7. Code integration test confirming system operates end-to-end in Arabic or Hebrew when Saudi Arabia and Israel locales are selected, respectively. System passes
   

Sprint Backlog Item with Tasks

In the second half of a Sprint Planning Meeting, the team breaks each Sprint Backlog Item into development Tasks, as in the example above. For each Task, one or more Developers volunteer, and those volunteers agree on the estimated work hours for the Task. Group estimation processes are rarely used for Tasks. There is no need to make sure the estimated hours in the Tasks sum to the estimated days for the Sprint Backlog Item. When the second half is complete, the Sprint Planning Meeting is over and developers begin their Sprint Work.

Sprint Planning Meetings should be time-boxed to 8 hours, typically 4 hours for the first half, 4 hours for the second. New teams or new backlogs may require all of this time. The Product Owner and ScrumMaster can make these meetings much shorter with Backlog Grooming.

Sprint Work

In Sprint Work, the Team starts tackling the work, beginning the highest priority Task in the Sprint Backlog, and moving down the list. A productive subset of Developers focuses on the topmost Sprint Backlog Item, taking on individual tasks. Another subset focuses on the next highest Sprint Backlog Item they can productively address (there may be dependencies or skill set issues), etc., until all Developers have something to do. If a Developer finishes all his or her Tasks, that Developer looks at the remaining work to see where he or she can help others, or move on to another Sprint Backlog Item. If no other Sprint Backlog work seems suitable, the Developer considers whether he or she can improve overall product quality, automate build processes, or otherwise improve the productivity of the team. As a last resort, a Developer might pull the next highest priority Product Backlog Item outside the Sprint Backlog, and attempt to complete it.

Each working day, the Team attends a Daily Standup Meeting (sometimes called a Daily Scrum or Scrum Meeting). Ideally, this meeting occurs at the same time every day and lasts no more than 15 minutes. Each Pig answers three questions: What did you do since the last Daily Scrum? What do expect to do before the next Daily Scrum? What impediments are currently blocking your progress? When all Pigs have answered, the meeting is done. Pigs can ask questions during the meeting, but, by virtue of the time-limit, answers must be kept very brief.

Sprint Burndown Charts

Sprint Burndown Chart

Sprint Burndown Charts help a Team track progress during a Sprint. The vertical axis is the total remaining estimated work hours for the Sprint, the horizontal access is work days. Every day, developers update the current Sprint's task list with the remaining estimated hours. If developers encounter difficulties their estimates can increase or if they add new tasks to a Sprint Backlog Item (both are allowed by Scrum), a task's estimated hours can go up. The ScrumMaster sums the remaining estimated hours at the end of the day and puts it on the graph.

In an ideal world, the Sprint Burndown Chart shows a straight line from the total estimated hours on the first Sprint day, converging to the X axis on the last day of the Sprint. It never happens; if it does, you should question whether the ScrumMaster is "cooking the data" to make it look perfect. Developers often discover estimated hours that were forgotten. When they are added to the estimates, the burndown line may diverge for a day or two.

Triage is a common activity mid-Sprint, for some Teams. If a Team discovers it cannot complete all the items in a Sprint, it revisits the Sprint Backlog with the Product Owner, makes sure that it can do enough backlog items to deliver a Feasibly Releasable Product, and moves low-priority items out of the Sprint (or leaves them in, but all agree they will be tackled only after everything else is done). Teams can also refactor Sprint Backlog Items to make them address part of a feature, preserving releasability, and add another Product Backlog Item to complete the job in a later Sprint.

Sprint Review

At the end of the Sprint, the Team hosts a scheduled Sprint Review Meeting or Sprint Demo. All stakeholders are invited. For each item in the Sprint Backlog, the team demonstrates, to the satisfaction of the Product Owner, that the Sprint Backlog Item was completed properly in the running product. In the best case, a third party installs the product following instructions written by the team, and the third party attempts to demonstrate each Sprint Backlog Item, thus showing that the product is releasable. In the worst case, the team asserts that it is done, with no validation. A responsible Product Owner in this situation should reject the Sprint Backlog Item, causing it to remain in the Product Backlog, to be later picked up in another Sprint. If all or most Sprint Backlog Items cannot be third-party validated, the Product Owner may consider sidelining or reconstituting the team.

Following the Sprint Review Meeting, the iterative cycle begins again with the next Sprint Planning Meeting. Note that during the Sprint, no one is allowed to change the Sprint Backlog without the unanimous consent of the entire team. However, Product Backlog Items outside the Sprint Backlog can be re-prioritized, rephrased or eliminated, and new Product Backlog Items can be added. This allows product development to adapt to new understandings of markets, product ideas, better assessment of Team skills, thoughtful assessment of risk, etc.

Sprint Retrospective

After the Sprint Review, the Team hosts a time-boxed 4-hour Sprint Retrospective to discuss and refine the process. This first involves taking pride in what was accomplished during the Sprint: What did we do right? Scrum is much harder than it may first appear, especially in dealing honestly and transparently with stakeholders and management.

The meeting also talks about improvements: What can fix, add or remove from our process that would achieve our goals better? Typical retrospective topics include revising the Done Criteria for Sprint Backlog Items, Tasks and Feasibly Releasable Products, finding more efficient ways to groom the Product Backlog, deciding how to split up Product Backlog Items better among multiple teams, considering new seating arrangements, deciding who can communicate a release forecast, etc.

Teams often spend 30 minutes to an hour listing possible improvements, then prioritize the list, then spend the rest of the time discussing specifics about how the highest priority items will be implemented in the next Sprint. Organizational impediments may block some improvements; if so, the ScrumMaster should work with managers to remove those impediments.

Release Forecasting

Velocity is a Team's historical average productivity in estimated-programmer-days per Sprint. The Sprint Backlog Items used to compute this number must have been completed in the Sprint where they are counted. If the Team estimates Product Backlog Items in advance, the team's Velocity can be used to forecast when any subset of those items could be completed.

Product Burndown Chart

A Product Burndown Chart displays a Team's Velocity, its progress in estimating future items, and the addition of new Product Backlog Items as the Product Owner and stakeholders gain experience with each Sprint's Feasibly Releasable Product.

The figure shows 100 estimated programmer days at the onset of Sprint 1. There were Product Backlog Items that had not yet been estimated. They were estimated during Sprint 1, and appear in the Sprint 2 bar, minus the estimates of Sprint Backlog Items completed in Sprint 1. From Sprint 2 to Sprint 3, you see the pure effect of the team's Velocity in completing about 22 estimated days of work. From Sprint 3 to 4, we see that the team burned down about 44 estimated days of work, but about 15 estimated days in new Product Backlog Items were added after the Product Owner showed the Sprint 2 feasible release to stakeholders. From Sprint 4 to 5, the team burned down 40 estimated days of work, but the Product Owner added 15 estimated days. At this point, we can project a possible release product at the beginning of Sprint 7.

Product Owners can consider how dropping different features from the Product Backlog might affect the release date. Of course, this must be done with thoughtful consideration of dependencies that might exist in the ordering of Backlog Items.

Challenges

To waterfall managers, Scrum is an alien process. To adopt Scrum is to admit that years of reliance on defined process planning—with its immediately-out-of-date Gantt charts and never-reached project deadlines—was wishful thinking all along.

Scrum places release forecast responsibility on the shoulders of Product Managers, who tune the release date based on the feature set they demand, and on empirical productivity metrics about the team. Using a waterfall process, upper-managers could blame their optimistic developers for a late release. But let's state the truth, it was usually worse: Most of the time developers had no chance to set a realistic schedule, rather a manager created an unrealistically aggressive schedule, perhaps in optimism or perhaps in an attempt to motivate productivity.

For developers, Scrum can be a challenging process. The term "Sprint" is very descriptive. I've had more than one developer state that "Scrum is intense." Scrum and other agile methods make it hard to hide sloth: developers have to report to the team daily, they have to release a working product monthly. Typically I've found that insecure people, rather than challenging the process itself, seek ways to limit transparency by modifying Scrum. They might suggest that Daily Standup Meetings occur only when needed, rather than daily, might find ways to avoid investigating the releasability of a sprint product, or might avoid test frameworks that can expose regressions.

Scrum challenges our job descriptions. Engineers become members of a highly-communicative, multifunctional team, and the team succeeds or fails together. The team is highly motivated to create automated unit, regression and integration tests. The team must determine "what is done", and hold itself to that standard. Does this mean the team is entirely responsible for QA? The answer to this question is different depending on the organization, but I can say, in Scrum, the team becomes gradually more and more involved in preserving and improving product quality. New projects have more flexibility; the team takes on QA tasks from its inception.

Group managers and directors have less-defined roles. They become change advocates, auditors, teachers. Sometimes they must get their hands dirty getting teams started, or making up for deficiencies that emerge. They must diplomatically push people in the right direction, making the process more efficient, keeping people happy and preserving their own integrity. To do decent performance reviews or allocate developers to teams, they must attend a lot of Daily Standup Meetings—if raises are misaligned with performance, it will be clear to the whole team.

With transparency comes unvarnished honesty and vulnerability. At one point, I was managing a Scrum-based project, but working for a waterfall manager. Every day was exhilarating, because my team led the way for others, and was scary, because I worried I would annoy people too much with my bluntness and get fired.

An obvious challenge: How do you scale this to a larger engineering group? Most all-Scrum departments use “Scrum-of-Scrums,” where teams aggregate all their product backlog items together into a giant list, and plan many backlog items together [4]. Sometimes they create and manage priorities around "Epics". I have never seen either approach used to manage multiple products. My company, Citrix Online, developed something different, which we call Enterprise Scrum. It handles multiple products, it is efficient, and it makes impediments and velocity visible at the highest levels of the company. You can find out more at my blog [5].

Summary

Scrum is an adaptive software management process that provides more visibility, more control, more accurate forecasting, and more productivity than standard waterfall processes.

A Scrum Development Team has three participant types: one Product Owner, one ScrumMaster and 3-7 Developers. Development proceeds iteratively, with fixed-duration development periods, called Sprints, lasting no more than a month. Each Sprint contains a Sprint Planning Meeting to determine what will be tackled in the Sprint, Sprint Work to complete development tasks, a Sprint Review Meeting showing off running code that could be suitable for release, and a Sprint Retrospective to tune the process.

Scrum includes three feedback-loops: Daily Standup Meetings help the team adapt rapidly and resolve problems together, Sprint Burndown Charts allow the team to review its progress and perform Triage to produce a working product in a short period. Sprints allows marketers and stakeholders to change development priorities on a monthly basis.

Product Burndown Charts provide the ability to forecast releases empirically, incorporating both developer productivity metrics and requirement addition trends into the forecasting process. They make planning and resource allocation easier. They provide a fourth feedback-loop, showing how product management choices affect release dates.

Jeff Sutherland assesses a company's agile maturity by asking CEOs these two questions: What are your teams' Sprint velocity? What organizational impediments have your teams identified, and what are you doing about them? To answer these two questions thoughtfully: your company must divide into Scrum Development Teams; each team must perform all the functions listed here; the teams must report velocity and impediments; upper-management must roll these data up for the CEO; and the company must be able to analyze and unblock team impediments. Simple questions are not always easy to answer.

As a software engineering manager, I've not found anything better than Scrum. I've worked in large and small companies, as everything from developer to CEO. In managing developer groups, I've long looked for a methodology I could apply with some rigor to deliver products on-time with a predictable feature set. I've used Rational Unified Process (RUP), Unified Modeling Language (UML), defined-process planning with Microsoft Project and Gantt Charts. None of these satisfied, though I have found some value in each.

In applying Scrum, I've seen greater productivity levels than I expected, and refreshing honesty and adaptability. It makes my work much more satisfying: I feel closer to customer needs, I think team members work at nearly peak performance. While we are typically exhausted at the end of a day, we also go home evenings and weekends and enjoy them.

References

1. Jeff Sutherland, Scrum Blog, http://scrum.jeffsutherland.com
2. Ken Schwaber, Agile Project Management with Scrum, Microsoft Press (2004), Amazon
3. Mike Cohn, Agile Estimating and Planning, Prentice-Hall (2008)
   Amazon
4. Mike Cohn
   Advice on Conducting the Scrum of Scrums Meeting
5. Dan Greening's Scrumerati Blog
   http://scrumerati.com
6. Anthony Finkelstein, Report of the Inquiry Into The London Ambulance Service (February 1993), International Workshop on Software Specification and Design Case Study
   http://www.cs.ucl.ac.uk/staff/A.Finkelstein/las/lascase0.9.pdf

Postscript

At the time this was written, Citrix Online was my employer, and supported my explorations and advocacy of agile management and marketing techniques. Many of its development teams are geographically distributed; we used Citrix Online's GoToMeeting product for our Daily Standup Meetings. Ron Lussier assembled the Scrum Cycle figure shown above.

This article was originally posted to Google Knol on October 30, 2008. It quickly became a top-pick, as well as a “Knol for Dummies” post. At the time of this post, it had over 12,000 page views. Google Knol is being decommissioned, so I have transferred this article to Scrumerati.com. If you are interested in Scrum and agile methods, particularly relating to enterprises or product management, I invite you to subscribe to Scrumerati.com by entering your email on the right side, or by adding its feed to your RSS reader.

Release Duration and Enterprise Agility: Global Webcast on 7 Feb 2012

Short release durations help companies test market theories to maximize profit. When managers measure release duration and advocate its reduction, they drive changes that improve the long-term productivity of the company.

Join me on February 7, 2012 at 11:00am EST, for a Scrum.org Global Webcast on release duration.  Register now at Scrum.org Global Webcast: Release Duration and Enterprise Agility.

We'll discuss how to obtain a release duration history from financial records, and how a finance-based definition can help you assess your ability to innovate profitably. We'll talk about various forms of technical debt and how they affect release duration.

 

Release Duration over a Company Lifetime

We'll show how measuring and restricting release duration helped improve code quality and processes in a mid-size software company. Both as a startup and mid-size subsidiary, the company's technical debt insidiously increased while it used traditional project management practices, even in the first months of its life, accumulating to a level where some projects took over 35 months to realize revenues. After focusing on reducing release duration and adopting Scrum and other agile processes, the company's engineering and product management groups eliminated much technical debt and drove release duration lower than it had ever been. The company's productivity, competitiveness and market share all improved during the same period.

Proposed: Agile Abandonment Disorder and its Root Causes

Jeff Sutherland, Scott Downey and I have found over the past few years that, more often than we wished, companies sometimes abandon agile methods. In our observations, it hasn't been because agile was performing badly; to the contrary, for the first time these companies had measurably improved their performance.

This year, Scott and I decided to tackle this issue as a research effort. We have looked at six companies, so far, but we hope to study more, with your help.

If you have observed agile abandonment, we'd like to talk to you. Your comments and company names will remain anonymous. My contact information is here.

If you've been working in agile coaching or training for a long time, and you've not experienced agile abandonment, or if you've found ways to counteract it, we'd also like to hear from you.

Scott and I proposed this topic for Agile 2012. Here is our submission:

Agile 2012 Proposal

Agile adoption can lead to dramatic improvements in market adoption, engineering efficiency improvements and engineer empowerment. But many companies have adopted agile practices, and then, despite seeing significant benefits, have abandoned those practices, allowed waterfall practices to return, and saw productivity decline. Why? If agile really helps companies succeed, how can we ensure its retention?

This session will summarize a series of root cause analyses we performed with agile leaders, using a “Root Cause Mapping” approach that helps groups of people identify contributing causes and share responsibility. Companies will remain anonymous in the presentation.

Most of us believe that agile adoption requires initial courage and effort, but agile retention also seems to require sustained courage and effort. Loss of a key leader, addition of a new leader, growth of the company beyond a certain scale, company acquisition, insufficiently refactoring mid-level manager roles, and the inability of product management to exploit agility all seem to correlate with agile abandonment.

Today, we have analyzed six companies that have abandoned agile methods. In discussing this issue with others, we have discovered abandonment is more common than we thought. We hope that this session proposal will inspire others who have experienced this problem to contact us with more examples. Email to dan+abandonment@greening.org.

If feedback is positive on this session proposal, we plan to incorporate additional companies, further analyze the phenomenon, consult with colleagues on mitigation strategies and assemble a set of best practices for protecting agile gains.

Help us out

If you think this topic merits discussion, consider lending your voice in favor of the topic. Registered users can provide public comments on the proposal at http://submit2012.agilealliance.org/node/14003.

Here's to persistent agility!

Could Your Company Lose Agility? Adaptive Portfolio Management can Help

Abstract

Agile methods create conflict. So to retain agility, we must actively promote it even after agile has taken hold. How can we equip a company with the cultural and process tools to sustain agility? Your choices may depend on your perspective. Social psychologists teach organizations to reinvent themselves, but this approach may be too slow for a large company. Change agents study, develop and evangelize specific processes, a more promising approach. "Adaptive portfolio management" is a specific process under development that may help sustain agile-thinking. It maximizes value by rapidly adapting to market and company changes. In using adaptive portfolio management, company leaders exercise agile fundamentals. They can become agile advocates for the whole organization, making agility more stable for developers on the ground.

In the service of productivity, agile methods create conflict. Consider continuous improvement, a canon of agility: It demands that people expose perceived problems and analyze their causes. Try it in any relationship, and you'll create conflict. Agile techniques increase transparency, increase accountability, weaken mid-level managers and deemphasize heroism: just a few more things that make some people mad. Some may start to see agile as the problem. “It was so much easier the old way!”

And so, agile culture requires active support even after it seems to take hold. Many experienced coaches have seen organizations that successfully implemented Scrum, and then stumbled. Some conclude this arises from software development's nature as a complex system. Scrum seeks to lightly control the chaos of software development—allowing for innovation while ensuring productivity; its feedback-driven control requires intellectual energy to sustain.

How can we equip a company with the cultural and process tools to sustain agility? The 2011 Scrum Coaching Retreat brought 70 Scrum coaches together in Boulder, Colorado Dec 7-9 to focus on this. We formed groups to analyse different elements of the problem.

Social psychologists and change agents

How we help a company sustain agile practices depends on our perspective. Some coaches are social psychologists—fostering agile culture and teaching organizations to reinvent themselves; others are change agents—studying, developing and evangelizing specific processes. I'm a change agent by nature.

I love analyzing and designing whole-company processes. The Agile Organization Design group seemed promising: it might have proposed ways to design organizations to support agile principles. However, social psychologists in the group challenged the presumption that we might design organizations at all. They sought strategies to enable an agile organization to design itself. One suggested that we "Fire all the managers." Another wanted the organization to "self organize". Self-organization takes a long time for Scrum teams, and much longer for large companies. As a result, leaders largely design companies by fiat. The change agents among us wanted to help those leaders, documenting organization designs that had worked for others, providing templates they might use as a starting point. The group made no progress; our goals were too different. I switched groups.

Adaptive Portfolio Management

The Adaptive Portfolio Management group provided a refreshing harmony, with likeminded folks seeking the best ways to prioritize, staff, interrupt and cancel projects. The group established a definition:

Adaptive Portfolio Management is the discipline of regularly changing the priority of business initiatives and resource commitments to maximize total portfolio return. This is implemented by:

1. Creating and managing an inventory of strategic initiatives
2. Prioritizing initiatives based on business drivers
3. Allocating budget to initiatives to maximize value
4. Periodically re-evaluating allocations based on changing needs
5. Transparently communicating across the organization

and a credo (really a manifesto, mimicking the format of the Agile Manifesto):

We value:

1. Optimizing the holistic portfolio over Protecting departmental budgets
2. Courage to change over Following a plan
3. Ability to evaluate the portfolio on a frequent basis over Infrequent (i.e. annual) planning cycles
4. Responding to emerging opportunities over Sticking to the plan
5. Maximizing value over Managing cost
6. Collaborating on decisions over Centralized authority

I was excited that 12 experienced people agreed on fundamental principles and worked so productively together. I was relieved that a portfolio management technique we developed at Citrix Online fulfilled the group's principles [gree2010]. We tried out an adaptive portfolio management game, developed by group member Jade Meskill [mesk2011]. In playing this game, competing teams largely developed the same risk-adjusted profit-maximizing strategies to order portfolio backlogs. We thought alike! Many of the graphics developed by the group would be suitable to describe Enterprise Scrum [scrp2011].

While adaptive portfolio management helps maximize a company's profitability, it can also help sustain a company's agile thinking and processes. If key decisions on project funding and prioritization are based on agile principles, those principles become infused in the corporate culture. Executives often own major product decisions. If those executives are using adaptive portfolio management, the rest of the company is likely to follow their lead and support other agile activities.

My colleagues and I are studying and developing better approaches for company-wide agile methods. We help companies become agile and stay that way. Contact me if the topic area interests you.

References

[gree2010]

Daniel Greening, “Enterprise Scrum: Scaling Scrum to the Enterprise Level,” 2010 43rd Hawaii International Conference on System Sciences (HICSS), Koloa, Kauai, Hawaii January 5-8, ISBN: 978-0-7695-3869-3 (10 pages)

[mesk2011

Jade Meskill. Agile Portfolio Management Game. Description and downloadable game materials. Blog entry. 9 Dec 2011.

[ries2011]

Eric Ries. The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, 2011.

[scrm2011]

Main Page, Scrum Coaching Retreat 2011, Boulder Colorado.

[scrp2011]

Adaptive Portfolio Management, Scrum Coaching Retreat 2011, Boulder Colorado.

Review: Liz Keogh, Learning and Perverse Incentives

Liz Keogh, Learning and Perverse Incentives, QCon London 2011

This 50 minute talk discusses perverse incentives: situations where incentivizing individual behavior causes an organization to become dysfunctional.  When we attempt to optimize an organization, but fail to use systems thinking (i.e., when we are optimizing from an internal perspective), we can create inappropriate results.

Different feedback mechanisms drive behavior in both positive and negative ways. Some examples: Reinforcing forces, balancing forces, feed forward, delayed feedback (and resulting oscillations). A positive feed-forward example: "Chance favours the prepared mind" (Louis Pasteur). If you prepare yourself properly, good things happen in the future. A negative feed-forward example: over-preparing for avoidance of a negative outcome. If many people worry about limited resources, people can hoard those resources and cause a systemic shortage.  Local optimization caused systemic failure.

We often place a strong emphasis on risk avoidance, but this can kill innovation. Risk is a natural part of any creative process, such as developing software, but some creative attempts will fail. When we take on risk, we learn. If you are in a creative field, you should never take on risk free projects. Some organizations make punishments so great for project failure, and rewards for doing new things successfully so minimal, that innovation stops. One perverse incentive is if someone punishes you for failing to achieve an estimate, you pad your estimates, and then your velocity goes up.  (See Liz's "Estimation Anti-Patterns" on her blog.) Another perverse incentive: requiring architectural review on after people on a team innovate and fail: too frequent reviews will kill innovation.

When we put on our Evil Hat, we say "How can game this system?"  That perspective is great for finding perverse incentives.  KPI's—Key Performance Indicators—are a potential source of gaming and arbitrage (exploiting that true value is different from the KPI). The quicker we're getting feedback from a metric, probably the further away we are getting from measuring true improvement. People tend to game the system for the metric, not the improvement, especially when KPIs are rewarded.

Your brain is brighter than you are. You will game the system, even if you think you aren't consciously wearing the Evil Hat. Rarely do people game the system deliberately, but there are some situations, particularly with middle managers, where people deliberately game the system for their personal benefit. (See Snakes in Suits below.)

How to improve:  Get away from solving a problem, and create a system where problems are solved. Evangelize and embrace frequent Root Cause Analysis, with 5 whys, to get to root causes.  Keep metrics close to the improvement you are trying to measure (see The Goal below.)  Create a learning environment. Even in standups, “This is what I learned today; This is what I hope to learn tomorrow; Can anyone help me find out... ?” could be useful.  Fit the process to reality, not reality to the process.  Try focusing on what's going right as well as what is going wrong. Then we focus on learning: what should improve, what should we keep? Thank people for doing a good job, and think “How can I anchor good work?”

Book List:

• Peter Senge, The Fifth Discipline: The Art & Practice of the Learning Organization
• Tom DeMarco and Timothy Lister, Walking with Bears
• Donald Reinertsen, Managing the Design Factory
• Edward de Bono, Six Thinking Hats
• Paul Babiak and Robert D. Hare, Snakes in Suits: When Psychopaths go to Work
• Eliot Goldratt, The Goal

Is Hiring More the Right Answer?

In my current company, I sometimes here people say “We need more people to handle this workload.” Every group in the company considered in isolation could use more people, no matter what group they are in; at least that's what most people think. But smart folks sometimes say, “We have too many people.”

Here's what the research tells us:  Until you reach about 7 people in a group, adding another person will likely make the group go faster [putn2011]. Specifically, starting from a 1 person group, at first, each time you add a person the value-produced per group goes up, However, something weird happens as you approach 7 individual contributors: The group productivity drops. I.e. An 8 person group produces less than a 7 person group. Why? Because the communication, coordination and mistake costs have overwhelmed the advantage of additional people.

To add one more complexity, the zero-sum game of corporate hiring means if your group hires more, my group must hire less. Smart software executives typically seek an optimal ratio of user experience designers, engineers, program managers, user experience managers, engineering managers, scrummaster, product managers, researchers, etc. to maximize corporate profitability. If your company has the right ratio, when a group is overwhelmed, you are actually over-stressing the whole system, producing technical debt, and should slow down.  

In my current company, user experience is one of our most important brand values.

Recently, a colleague asked me to consider hiring more user experience designers for a product where software engineers develop new features very slowly. Given that insight, you might suspect the software carries a lot of technical debt, and you would be right. If we hire more user experience designers, it would likely generate more feature requests, then more pressure on the engineering team to produce features faster, then more shortcuts in coding, and then even more technical debt.

Technical debt leads to systemic slowdowns in productivity. Its accumulation is insidious and can be disasterous for a company. In the face of a highly competitive marketplace, technical debt scares me.

Here are some sophisticated alternatives to consider when a user experience group is overwhelmed:  

• The ratio of user experience people to developers really might be too low. In this case, reduce the rate you hire engineers to increase the rate you hire user experience designers.
• Engineers might be producing features too fast; you should get more engineers to work on reducing technical debt to make the future brighter.
• Your user experience group might not be saying "No" to unimportant assignments. In some cases, product owners or engineers can do a reasonable job with a little guidance. In other cases, the project isn't profitable enough to bother doing.

You can apply this sort of thinking to any situation where you think you need more people. Will more people make your group more productive?  Even if the group is more productive, will the company be more productive? Take a step back and think about the relative ratios of different types of contributors. Maybe the company would benefit most from adding more people to a different group.

When someone asks you to state your team's top impediment, here is what they are asking: “What is the biggest thing keeping my existing team from contributing most to corporate productivity?”  Hopefully this post explains why “Not enough people” isn't usually a good answer.  Sometimes, in fact, the impediment is too many.

[putn2011]

Doug Putnam, “Team Size Can Be the Key to a Successful Software Project,” Quantitative Software Management, IT Almanac, 2006.

Quality is Not Negotiable? A Contrarian View

In Tradable Quality Hypothesis, Martin Fowler says “as soon as you frame internal quality as tradable, you've lost.” At the risk of being rode out on a rail, I argue that making quality non-negotiable lets engineers off the hook for understanding internal quality. Quality is always negotiable, and you should understand the benefits and costs of internal quality so you can negotiate and prioritize well.

Quality has gotten short shrift in most organizations. Software quality can dramatically affect future productivity. Since we often inappropriately sacrifice quality for features, it can sometimes make sense to stamp your foot and say "quality isn't negotiable."

However, understanding which software quality efforts actually matter requires us to think more deeply. How does software quality affect us? Most software quality problems—copy-paste code, non-standard formatting, bad encapsulation, poor use of modern language features (generics, iterators, etc.), simultaneously maintained code branches—degrade future engineer productivity. Other software quality problems—known bugs, inadequate test coverage—degrade software value.

With a subtle understanding of quality, you can answer daily questions in context. For example, should you refactor a long-untouched body of code for future productivity? Sometimes you shouldn't. But if fear of touching that code holds back new functionality, maybe you should. The Tomcat 7 committers just refactored a lot of long-untouched code, with likely productivity improvements going forward (and a whole bunch of bugs fixed). [see Mark Thomas: Introducing Tomcat 7 at 39:30]

Unless we make engineers defend their quality efforts, they fail to thoughtfully prioritize quality improvement efforts. Should they create unit tests or integration tests first? Most engineers can't give you a thoughtful answer to this question. But they are going to do something first, and that something might be a lot less valuable than the other thing. I've seen poorly conceived quality efforts squander millions of dollars.

That's why "quality is non-negotiable" articles always fall flat with me. They punt the tough questions. They seem well-suited for the "I never think about the value-side" engineering people to deal with "I never think about the effort-side" product people who keep making one-sided demands.  

I like using Net Present Value to think about the cost-savings of quality improvement and the value of features. It helps me prioritize and compare options. Engineers have to make decisions about quality every day. They should hone their decision-making skills, and clichés are not likely to help.

Review: The World is Not Small for Everyone

Many decades ago, my small-world assumptions were incredibly optimistic: I was smart, resourceful and helpful, and the world needed me, therefore capitalists would seek me out and pay me well. I discovered they weren't looking very hard for me: I had to find them. I longed for experts in my own social group—i.e., gay software entrepreneurs—who could advise me, and found few. I even wrote to venture capitalist colleagues asking if they knew any gay VCs: nope. I remember thinking I couldn't talk football and I didn't play golf, therefore I would find few commonalities with straight men. Though I now realize how prejudicial this assumption was, it turns out the main victim was likely me.

New research on the “small-world theory,” that we can find what we need in few steps via our social networks, shows that preferring advice from members of the same social group may damn us to more difficult access to expertise and referrals.

In “The World is Not Small for Everyone” [sing10], Singh, Hansen and Podolny thoughtfully explore how limited access to knowledge hobbles peripheral, junior and female employees. Surprisingly, in all three of these cases, self-reinforced isolation seems to interfere further with access to experts and referrals.

Knowledge is the primary resource in management consulting. It's all about who you know. When a client raises a question management consultants can't answer, they look for an in-house expert. If they don't know an in-house expert directly, they ask a colleague for a referral. The sequence of referrals ultimately forms a search chain. All else being equal, the shorter the chain, the more rapidly a management consultant can offer an answer, the more likely the client will appreciate the consulting engagement, the more engagements the consultant might obtain, and ultimately the higher the billing rate the consultant might command.

Singh et al surveyed 3,150 consultants in a a global management consulting firm to test these hypotheses:

1. That searchers who have low expert-related centrality, have short tenure, or are in the gender minority are less likely to pinpoint an expert in the first step of a search chain,
2. That searchers are more likely to select intermediaries with whom they share a characteristic (expert related centrality, tenure, gender) than intermediaries with whom they do not.
3. That searchers (original and intermediaries) who have low expert-related centrality, short tenure, or are in a gender minority tend to have longer realized search paths.
4. That for searchers who belong to the organizational periphery, those who name members belonging to the core as the next search step tend to have shorter realized search paths than those who do not.

All of these hypotheses appear to be true.

Women, peripheral employees and new employees all suffered longer searches in finding expert knowledge. Of the three, women were the hardest hit. People in these disadvantaged groups who crossed social boundaries to contact people outside their group reduced their disadvantage significantly.

This paper seems to confirm an advantage of cultural assimilation: by crossing social group boundaries more frequently, members of disadvantaged groups can establish stronger connections with experts and well-connected networks, thus improving their economic standing.

Agile management methods may mitigate some of the disadvantages discussed in “The World is Not Small for Everyone.” Agile methods encourage the creation of cross-functional teams [vodd10], making it easier to obtain knowledge and referrals from people in different social. Agile methods encourage integrating senior and junior people in teams: With pair-programming, senior employees build stronger bonds with junior employees, bringing junior employees closer to the expert network [melo03].

At the same time, this paper challenges us to think more deeply about organizational structures, and how they reinforce or diminish social boundaries. I have recently seen situations where disadvantaged employees sought referrals from people in the same cohort, despite their awareness of others with better connections or greater expertise. How can we encourage social boundary crossing, to improve organizational knowledge and profitability?

Finally, it challenges us, as individuals, to introduce ourselves to alien social groups, and to overcome our natural tendencies to seek expertise only from those in our own social groups. The rewards may go beyond the economic: we may find friends in unusual places.

A few years ago, I attended a 25 year high school reunion and participated in the golf outing, despite knowing nothing about golf. The former captain of the football team taught me how to play. I had a blast.

With a slightly tilted perspective, I remain your humble servant. —Dan Greening

References

[melo03]

Thomas Meloche, James Goebel and Richard Sheridan, “Paired Programming in the Software Factory: Questions and Answers,” white paper, The Menlo Institute LLC, 2003. http://www.menloinnovations.com/freestuff/whitepapers/paired_programming_q_and__a.pdf

[sing10]

Jasjit Singh, Morten T. Hansen, and Joel M. Podolny, “The World Is Not Small for Everyone: Searching for Knowledge in Organizations,” Management Science 56:9, September 2010,pp. 1415–1438. DOI: 10.1287/mnsc.1100.1201 http://mansci.journal.informs.org/cgi/content/abstract/56/9/1415

[vodd10]

Bas Vodde, “Feature Team Primer,” http://www.featureteams.org/feature_team_primer.pdf

Learning Science and Agile Training

New cognitive psychology results can help us provide better training. Trainers seek to transform the way you think about tasks, motivation, planning and outcomes, and equip you with enough understanding to succeed. My Scrum Trainings are done in the afternoon, reinforcing learning by exploiting sleep cycles. Further ideas include changing venues from day-to-day, varying ways of applying agile thinking to problems, etc.

As Director of the Agile Program Office at Citrix Online, I train people in agile thinking, including XP, Scrum, Lean and enterprise-level productivity improvement. I'm keenly interested in approaches that enhance learning, especially in Scrum Training. Agile methods are difficult for many to fully embrace, and I want to do anything I can to help.

Scientists have known for years that sleep enhances memory [elle2006] [biel2006]. Traditional trainers provide Scrum Training as two solid 8-hour business days of classes. At Citrix Online, I conduct Scrum Training over four afternoons, with 4-hours each day. I prefer afternoons, to replicate studies that seem to indicate proximity to sleep provides for better learning. [scul2010]

This afternoon approach may provide for better conceptual learning, but it also has other advantages. By spreading out the training, it gives the trainer time to adapt to the interests and experience of attendees. Co-trainer Nick Kim and I used 4-day training to better adapt our team-teaching. Ultimately, we have alternated short lectures and exercises among each other. This made for a more diverse teaching style, and for livelier exercises.

By leaving students with mornings free, training was less disruptive to their work. They had a chance to coordinate with others in the morning and feel more focused in the afternoon. In addition, routine work in the morning allowed students to think about how Scrum, Agile and Lean methods might change how they work.

Research on learning also indicates that some commonly accepted truths are false and suggest surprising approaches worth considering [care2010]. For example, catering to left-brain or right-brain dominance in students apparently has had little value in helping them learn [pash2008].

In contrast, varying the context in which learning takes place, including changing the actual room, can help increase conceptual memory [smit1978]. So, perhaps, I will hold the next Scrum Training I provide in different classrooms on different days; we need to mix it up a little, so the ideas will gel.

Is your trainer applying these techniques when they teach you?

Though an occasional reference section arises vestigially from my academic past, I remain your devoted servant in the present. —Dan Greening

References

[biel2006]

David Biello, “Snooze or Lose: Memory Retention Enhanced by Sleep,” Scientific American News (July 11, 2006). http://www.scientificamerican.com/article.cfm?id=snooze-or-lose-memory-ret

[care2010]

Benedict Carey, “Forget What You Know About Good Study Habits,” New York Times (September 8, 2010). http://www.nytimes.com/2010/09/07/health/views/07mind.html

[elle2006]

Jeffrey M. Ellenbogen, Justin C. Hulbert, Robert Stickgold, David F. Dinges and Sharon L. Thompson-Schill, “Interfering with Theories of Sleep and Memory: Sleep, Declarative Memory, and Associative Interference,” Current Biology, Volume 16, Issue 13, pp 1290-1294, 11 July 2006, doi:10.1016/j.cub.2006.05.024. http://www.cell.com/current-biology/abstract/S0960-9822%2806%2901607-1

[pash2008]

Harold Pashler, Mark McDaniel, Doug Rohrer and Robert Bjork, “Learning Styles: Concepts and Evidence,” Psychological Science in the Public Interest 9:3 (December 2008), pp. 105–119. http://www.psychologicalscience.org/journals/pspi/PSPI_9_3.pdf.

[scul2010]

Michael K. Scullin and Mark A. McDaniel, “Remembering to Execute a Goal: Sleep on It!” Psychol Sci. 2010 Jul;21(7):1028-35. Epub 2010 Jun 2. http://pss.sagepub.com/content/early/2010/06/02/0956797610373373.abstract

[smit1978]

Smith, S. M, Glenberg, A. M., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342-353. http://bjorklab.psych.ucla.edu/pubs/Smith_Glenberg_Bjork_1978.pdf

Meeting Scrum: Part 1, Make Information Flow

Concerned citizens might warn you not to stand to close: I rope nearby bystanders into the crime I'm currently committing. So when Scott Downey, the General Patton of Hyperproductive Scrum, told me he was busy crawling wineries near my Santa Barbara office, I suggested getting together. “How about 11:30am for lunch?” I asked. “Perfect,” he said, “See you then.”

Scott arrived, we had a hearty repast, and after agreeing the world needed our genius, I had to get ready to to teach the second afternoon of Scrum Training at Citrix Online. Did he want to come? “Sure,” he said. As we walked into the conference room, I asked if he might like to present some of my slides? “Why not?” he said. When we started looking at them, Scott felt uncomfortable. Could he use his own slides?

Figure 1. The General Patton of Hyperproductive Scrum

Yes, he could use his own slides, and he did. In 20 minutes, Scott and I eliminated concepts I had already discussed from his very different Scrum Training deck, and restructured slides that would be relevant for that afternoon into short blocks. Scott trained the entire afternoon solo, with me coordinating and heckling. The transition was nearly seamless. Our students liked our surprise guest, and our sparring interaction helped spice up the material.

To make this possible, I relied on a highly-adaptive meeting format I call Meeting Scrum. It can turn dreary, boring meetings into vibrant, audience engaging events. The next time you have a half-day meeting, you might give a try.

Think Iteratively

Meeting Scrum decomposes high-impact training programs and full-day meetings into  digestible chunks. It involves your audience to help reconfigure information and agendas  on-the-fly to adapt to their background, your skills, last-minute opportunities and plain old serendipity.

Meeting Scrum miniaturizes concepts from Scrum, a software management technique. Scrum produces software in short fixed-time iterations called “sprints,” and demands each sprint produces software that customers value. This requires extraordinary discipline, but typically delivers high-quality software to customers faster.

How long should Meeting Scrum sprints last? Whether in Meeting Scrum, Software Scrum or Enterprise Scrum, sprints should always be long enough to provide independent value, and short enough to allow for rapid continuous improvement. One hour isn't always enough, especially when learning games are involved; more than a couple of hours means you won't be able to adapt to feedback even in longer meetings.

Two-hour sprints have worked well for me. Attendees can learn valuable skills and provide useful feedback after two-hours. Agile Training at Citrix Online is 16 hours long, providing 8 two-hour sprints. All-day meetings are 8 hours long, providing 4 two-hour sprints. Even in half-day meetings, you get a chance to change the agenda mid-meeting.

Create a Topic Backlog

Before using Meeting Scrum, prepare your program for iteration. Every program can benefit from an agenda. If you are starting with a giant, unstructured Powerpoint deck, decomposition for Meeting Scrum will improve your presentation.

Work items in Meeting Scrum are called “topics”. Give each topic a clear title, put it in a document or spreadsheet, and then rank-order the list. Of course, cover basic topics first; as you build understanding, cover more advance topics.

If this is a teaching situation, break up lectures with exercises: direct, multi-sensory experiences help people learn. Each exercise is a topic of its own. Remember: you don't have to explain a concept before you do its exercise. Sometimes an audience frustrated by an early exercise is more open to lectures that follow.

Your list of rank-ordered topics and exercises is called a “topic backlog.”

Groom Topics

Excessively broad topics take too long to keep an audience's attention. You should be able to cover 3 or more typical topics in two hours. If a topic is too long, break it down into smaller topics and then wrap them up in a final topic. You don't have to cover all these topics in a single sprint.

How do you know whether a topic will be too long? If your program is slide-driven and you are an experienced presenter, you may already know your approximate slide velocity (slides per hour). I can do about 8-15 slides an hour in Agile Training. I don't like my topics to be longer than 8 slides long. Open-discussion topics and exercises require some estimation. I estimate a non-slide topic in “slide points,” roughly how much time it might take in slide-equivalents.

Figure 2. Restructuring Scott's “My First Sprint” Topic

We had to groom Scott's program in 20 minutes. On the first training day, I had already covered the motivation for agile and productivity fundamentals. We pulled Scott's motivator slides out. We agreed that his “My First Sprint” slides would be a good next topic. But at 49 slides, it had to be broken down. Figure 2 shows the hierarchy we created. In the process, Scott reordered slides to fit into these topics.

Iterate

We then started the two iterations for the day, Sprint 3 and 4. They had this schedule:

Sprint 3	Sprint 4	Activity
1:05pm	3:05pm	Plan
1:10pm	3:10pm	Work
1:50pm	3:50pm	Questions
1:55pm	3:55pm	Work
2:40pm	4:40pm	Review, Retrospective
2:50pm	4:50pm	Snack

In later posts, I'll describe how we structured these sprints, the ultimate outcome of our Scrum Training, and how Meeting Scrum has changed since we started using it.

Challenge Questions

Some questions to think about:

1. How many slides do you think Scott completed in Sprints 3 and 4?
2. If you are familiar with Software Scrum, who are the stakeholders in Meeting Scrum?
3. Who is the Product Owner in Meeting Scrum?
4. What might transpire in a Meeting Scrum standup?
5. What rules might we apply to decide whether a topic could be started (the Topic “Ready Criteria”)?
6. What rules might we apply to decide whether a topic was completed (the Topic “Done Criteria”)?

With a tendency to stretch and shrink things, I remain your humble servant.