Scrum is an agile software project management method, though it could be used with some adaptation to managing projects of other kinds. "Scrum" is not an acronym, it just reflects the coordinated yet messy and noisy progress that a team can make.
Origins
The "relay race" approach to product development ... may conflict with the goals of maximum speed and flexibility. Instead a holistic or "rugby" approach – where the team tries to go the distance as a unit, passing the ball back and forth – may better serve today's competitive requirements.
- Hirotaka Takeuchi and Ikujiro Nonaka - "The New New Product Development Game" – Harvard Business Review, January 1986.
Scrum was popularised mainly by Ken Schwaber and Jeff Sutherland in the 1990s, based on earlier Japanese work. In the 2000-and-noughts, it is becoming popular.
What's the basic idea?
"A complex system that works is invariably found to have evolved from a simple system that worked."
Gall's Law
Scrum attempts to overcome the shortcomings of traditional, waterfall software development:
- Upfront design does not cope well with change - it assumes falsely that the users know exactly what they want, and hinders new ideas or discoveries about what would be good.
- Changing business circumstances on software long projects can make the software obsolete before it is deployed.
- Late integration and testing means that bugs and design errors are much more costly.
- But if change is allowed, undisciplined change can lead to chaos, poor quality software, and is avoided by software contractors due to disputes over cost and increased effort.
- "Big bang deployments" where the system has never been used in production on day zero, and is expected to function perfectly for a large user base on day one have a very poor track record.
All of these issues arise from the sequential gates that software development goes through: first requirements are completed, then all design is done before all coding, all coding is done before it is "thrown over the wall" to the testing team, and so on. I doubt that anyone actually works exactly this way; it is impossible to do it, there is always to-and-fro between the activities.
Scrum does the following
Business Guy: "You mean, with Scrum, I don't have to wait 18 months to get software I don't want?"
Ken: "No, We'll give you software you don't want in 30 days."
- An anecdote from Ken Schwaber.
- Deliver software iteratively, in relatively short cycles of a few weeks. These are called "sprints". If the work can't be finished in the sprint, do not extend the length of the sprint, rather hold some of it over for the following sprint.
- Deliver complete slices of software. Each sprint delivers a small slice of software with all activities complete - from design through to testing. You don't have to wait until the project is complete before getting some benefit from the software development work. If the project is 10% complete, you should be able to get 10% of the benefit from it right now. In practice, it could be more like 80% benefit from 20% of the software, since high-priority items are done first.
- Cross-functional teams. Rather than a design team that finishes and hands off to coding team, the scrum team includes all people currently working on the project. All activities happen in parallel. In typical software development, this can including coding (of various flavours such as web front end, back end and database), software testing and deployment, business analysis, user experience and interface design. The team is autonomous and in close communication (co-located if at all possible).
- Manage change: What the software is supposed do, the requirements, is not to be changed during a sprint. However after a sprint, when planning for the next sprint, any action is on the table, from continuing as before to abandoning the project, redesigning, or moving to a live deployment. But the usual action to re-evaluate the work priorities and commit to doing the items that are now highest-priority.
- Prioritize work: A prioritized backlog of requirements is kept. Requirements do not all have to be specified in detail before any development work can start. Only one sprint's worth of work needs to be fully detailed before the sprint starts. The rest will be less and less detailed they trail off into the distance.
- Inspect and adapt Communication is good. Visibility is good. Raise and react to problems quickly. Inspect and adapt the process used and the software created after every sprint, and the project goals as well. Inspect and adapt the priorities of remaining work after each sprint. Inspect and adapt the work being done in a daily meeting.
What scrum is not
Scrum is not "high ceremony" and rigid. It is an empirical and light-weight methodology, which aims at getting results. It should encourage the software development team to take their own decisions rather than have them dictated to them by the method. Scrum will not make software design decisions for you. And it will not save you from bad design decisions.
Scrum is not a process that specifies what to do for success at every step. It is a method for generating process, and sometimes for discarding process when it's more overhead than it's worth.
Scrum will not solve all of your problems. At best, scrum will just make problems visible through increased communication. What happens after that depends on your organisation.
Scrum will not turn idiots into software superstars. It can unleash the talent, if the talent is there in the first place. The daily scrum will at least give everyone a better idea of how much value each person is adding.
Scrum will not guarantee success. But if it's done well, the increased feedback will let you at least fail fast where you would have failed slowly without it.
Scrum is possible to do wrong. In fact it's easy to do it badly, if approached without discipline.
Scrum is not undisciplined. In fact all agile methods are quite strongly disciplined . They all require strict discipline to adhere to practices such as not breaking the build, no code without unit tests, keeping scrums down to 15 minutes, not changing requirements during sprints, listening to sprint feedback, maintaining the burndown chart, etc.
Scrum doesn't mean that any aspect of the required software can be changed at any time. In fact, this can only be done in sprint planning. The rest of the time, this is not allowed.
Scrum does not mean that there is no project documentation. Requirements can be consumed by a scrum team, or design documents generated if you insist; it just happens one sprint at a time.
Scrum is not a software development method, it is a project management method that is suitable to software development, and other exploratory activities. It is compatible with Agile software development methods such as Test-Driven Development and Extreme programming, and can be used alongside them.
Scrum is not all that complicated. You can learn it in a week. The rest is common sense and feedback.
Activities
The daily scrum
The daily scrum is a meeting of all people in the team. It is performed standing up, and should last 15 minutes or less.
You generally have three things to say at a scrum:
- What you did yesterday (or since the last scrum)
- What you aim to do today
- Impediments: what obstacles stand in your way.
A scrum is not the team being told what to do for the day, it is conversation between the members of the team, where everyone tells everyone else what they are doing. Scrum is not intended to exhaustively discuss and solve your obstacles, it is intended to raise them so that they can be discussed with the right people afterwards.
Anyone can come to the scrum, but only those committed to the project have something to say. There is a standard story about this:
A chicken and a pig are together when the chicken says, "Let's start a restaurant!".
The pig thinks it over and says, "What would we call this restaurant?".
The chicken says, "Ham and Eggs!".
The pig says, "No thanks, I'd be committed, but you'd only be involved!".
The "pigs" or participants in the scrum are the team, ScrumMaster and product owner (this last one is debatable). The company CEO is welcome to sit in as a "chicken", but he's not a "pig".
If there are too many people on the team for the scrum to work effectively, break it up into smaller teams with their own scrums.
Scrums typically happen at a fixed time, towards the start of the day.
The sprint
The sprint is an iteration of the project, a cycle in which software is produced. In each sprint the team will gather requirements, design, build, test and deliver software.
The length of a sprint is generally set for a project at somewhere from 1 to 6 weeks.
How long should your sprint be? If your project is e.g. experimental, involves rapid prototyping and frequent changes, then it could be as low as one week. If your project is large, or involves relatively fixed requirements for a client who knows well in advance what they want, it could be as long as a calendar month. Six weeks is generally (but not always) too long. If the business has a well-defined business cycle of a month, then you may want to synchronise with that.
What if you can't deliver what you wanted to in the time given? The deadline does not move, the sprint is not extended. Things that did not happen in the current sprint will be carried over to the next sprint.
All activities should happen each sprint - the software should be shippable at the end of each sprint. It may not be shipped, and "going live" may involve a sprint in itself, but you should not let "integration work" and a backlog of bugs build up.
Scrum breaks this down the waterfall by insisting on small iterations, but the sprints are not "mini-waterfalls" with distinct phases, rather the work is overlapping. If front-end developers depend on work done by back-end developers, then the back-end developers will be working up to a sprint ahead of them. The closeness of the team should increase collaboration in reduce the costs of inevitable change.
Sprint Review and Sprint planning
At the end of the sprint, there is a sprint review which presents what happened and looks back on what went well and badly. At the start of a sprint, sprint planning starts off the next one. In practice these generally happen back to back, since the sprint ends and the next one starts. How long to spend on these? I have typically seen one day for the two, but it depend on how long your sprint is. If you have month-long sprints, you could devote a day or two without too much drag. On a one-week sprint, a lot less than that.
Roles
ScrumMaster
A member of the scrum team who removes obstacles, ensures that procedures are followed, keeps charts updated, shields the team from external influences and generally keep the team productive however they can. They must know scrum, and are the closest that scrum comes to having a manager role.
I have seen good and bad ScrumMasters, and in my opinion the good ones have a certain level of forceful personality. They are able to enforce the rules and tell people what needs to happen once in a while. They are the ones who must interrupt and tell people to "talk about it after scrum". But they are also not micro-managers. They need to let people get on with solving problems in a self-organising way.
e.g. bad practice:
Mike: Yesterday I was trying to test the TPS reports, but I was having problems connecting to the departmental database. I'm getting an odd error. Today I will continue trying to fix that.
Pete: Is that like the Transitive trust issue that I had last month?
Mike: I didn't know you'd come up against that before. I tried establishing a trust relationship, but nothing changed.
Pete: That won't work unless you apply a patch.
Mike: Ok, where can I find the patch?
... 5 minutes of technical discussion later, everyone else is wishing they were somewhere else and subconsciously forming the opinion that scrums suck.
Better, the ScrumMaster gives some guidance:
Mike: Yesterday I was trying to test the TPS reports, but I was having problems connecting to the departmental database. I'm getting an odd error. Today I will continue trying to fix that.
Pete: Is that like the Transitive trust issue that I had last month?
Mike: I didn't know you'd come up against that before. I tried establishing a trust relationship, but nothing changed.
ScrumMaster: Ok guys, you can sit together and sort it out after scrum.
Mike: all right.
Best, the team have internalised the basic rules, and the ScrumMaster gives other guidance:
Mike: Yesterday I was trying to test the TPS reports, but I was having problems connecting to the departmental database. I'm getting an odd error. Today I will continue trying to fix that.
Pete: Is that like the Transitive trust issue that I had last month?
Mike: Yes ... but we can talk about it afterwards.
ScrumMaster: Good. If this is recurring issue, don't forget to put a note about it on the project wiki.
Product Owner
This is the person who says "you are making this software for me and I want it like this". They are the contact point for the vague, changing, contradictory and naive set of wants that we call "software requirements". They may not even be a user of the software, but they do represent them. For instance, for point of sale software used by store clerks, they will not be a store clerk but a manager of store clerks. However a sample user can be brought in to discuss and test actual usage. When software is sold or on the internet, the Product Owner may come from a marketing department.
The product owner establishes the vision - the concept of what the software is, should have some idea about what the return on investment is, takes the lead in populating the product backlog, has the final say in prioritizing it, and decides when a sprint should be released.
The product owner does not have to attend every scrum, but if they never or seldom attend this is likely to be a symptom of a problem.
Team Member
The scrum team is self-organising. In theory, all team members are equals and work together, picking up work as needed. In reality, you all have different skills. In best practice, your skills rub off on each other from close proximity.
Other artefacts
Product backlog
The product backlog is work to do in the entire software project. It does not need to be complete, you only need to be able to find out what to do in the next sprint. The items in it will be sorted by priority, and the high-priority ones will be detailed enough to put them into a sprint backlog. This accords well with reality of software - we will have some things that we know we want know based on what we have, some less-well defined things that we'd like to have later, and some vaguer, far-away wishes.
Sprint backlog
The sprint backlog is work to do during the current sprint. It is planned in sprint planning.
Burndown
A "burndown" or "burndown chart" is common- this graph tracks progress over the sprint, showing how much work remains to do over time. The rate it which this decreases is the "velocity". If the line, trended outwards, will not hit zero before the end of the sprint, then you have probably over-estimated the amount of work that the team can handle in one sprint, and one or more tasks will have to be removed from the sprint. If you are lucky enough to have it extrapolated to hit zero well before the end of the sprint, then more work can be added.
Summmary
There's lots more to say about Scrum, and you can find books and courses on that. As Methodologies go, scrum is quite simple. You can learn it in a week. Beyond that, it relies on the application of common sense, albeit based on some hard-won realisations about the changeable nature of software development. This is its strength.
References:
Hirotaka Takeuchi and Ikojiro Nonaka - "The New New Product Development game" Harvard Business review, January 1986
Ken Schwaber, Mike Beedle: Agile Software Development with Scrum, 2002
Wikipedia: scrum: http://en.wikipedia.org/wiki/Scrum_(development)
DeGrace and Stahl, "Wicked Problems, Righteous Solutions" 1991
Sources: Experience, co-workers, notes from Mike Cohn's Certified ScrumMaster course.