“No battle plan survives contact with the enemy.”
— Helmuth von Moltke
Sooner or later you will have the business asking questions like “How much is it going to cost?”, “What will I get?” and “When can I have it?”. The honest answer to this which often provokes a reaction of surprise are: “How much do you want to spend?”, “Tell us what you want” and “As long as it takes” respectively.
Technology projects are complex and high risk, and in particular there is high uncertainty especially before any actual development taking place. The Cone of Uncertainty [McConnell] describes the uncertainty in estimates of software development project cost, effort, and duration, throughout the lifecycle of a project. The degree of uncertainty is very large at the beginning of the project, which is often when decisions are made about project pricing and contracts which results in unrealistic project plans and expectations.
Agile or traditional, estimates on duration and costs are inherently very vague at the beginning of a project. For some complex projects, this uncertainty can last much later in the development process.
We are so accustomed to fixed price, time & cost projects, but in reality they are anything but. None of this information is new. However, project sponsors consistently latch on to the initial estimates project managers propose for the schedule and cost of a project at the start when the least is known and when there is the highest variability. And because project sponsors don’t understand project complexity and other factors influencing cost and schedule when requirements change and emerge, they may see the project as a failure if costs increased, even if the changes improved the value delivered to the business. Unfortunately, business executives’ lack of understanding about project management influences their perception of IT project success and failure [CIO].
There is inherent uncertainty, even in well-run projects. This is especially true in projects that involve complex software, large systems, large development effort, new application domains, new users. Furthermore, unlike manufacturing or building a house software development is a creative process and largely knowledge work which does not fit a cookie cutter model. We need to recognize that the Cone of Uncertainty implies substantial risks for traditional fixed-price and fixed-scope projects. The Cone of Uncertainty indicates we cannot predict the future accurately and that we need to restrain from traditional thinking. If you can’t predict the future, don’t plan it in detail.
A barrier to agile is the lack of the comforting predictive detailed plan whereby we plan out the uncertainty in a project, plot the path, determine the end date and the cost then produce the nice reports along the way. Interestingly enough whenever we question “are those plans ever right?” the response is often negative. Yet we place so much faith in them and they absolutely have to be there because it would be too scary to proceed without one.
Agile practitioners will do an order of less magnitude less effort to come up with a plan and provide answers to the questions “How much is it going to cost?”, “What will I get?” and “When can I have it?” that are just as good and just as inaccurate as traditional approaches.
We strive to ask for exact answers when exact answers don’t exist (estimates are just an approximate judgment or calculation – it’s a guess, but we seem to forget that). Reality will come to light soon enough. Striving for exact answers typically results in both time and money being wasted with little improvement in accuracy of the estimates to show for it.
An agile project will come up with a baseline view of how much the project will cost and when it is likely to be finished. The only difference to traditional thinking is that agile is very explicit right from the start that this is not accurate and the team will strive to understand more as soon as we start working it. We want to get to the actual development effort where we regularly deliver high quality, working software that maximizes the business’ ROI.
There is an amazing feeling of relief when agile teams can become comfortable with uncertainty and a freedom to actually discover what is needed at the appropriate time. Instead of relentlessly following the pre-defined plan agile teams will adapt to the inevitably changing conditions.
Being on time, on budget and to specification means nothing when we have delivered the wrong product with low quality that has no business value.
So rather seeking the exact answers, should we be asking if we are delivering business value?
In the book Switch: How to Change Things When Change Is Hard by Chip Heath & Dan Heath, the authors described the results of a study to answer the question: Would people with bigger popcorn buckets eat more at the movie cinema?
As part of the study, it was carefully engineered to serve the popcorn five days stale. Some people received medium sized buckets and others got a large bucket. The results were stunning:
People with the large buckets ate 53 percent more popcorn than people with the medium size. That’s the equivalent of 173 more calories and approximately 21 extra hand-dips into the bucket.
The book further states that other popcorn studies were always the same:
It didn’t matter if our moviegoers were in Pennsylvania, Illinois, or Iowa, and it didn’t matter what kind of movie was showing; all of our popcorn studies led to the same conclusion. People eat more when you give them a bigger container.”
No other theory explains the behavior. These people weren’t eating for pleasure. (The popcorn was so stale it squeaked!) They weren’t driven by a desire to finish their portion. (Both buckets were too big to finish.) It didn’t matter whether they were hungry or full. The equation is unyielding: Bigger container = more eating.
Recently in To Change Culture, Change the System David Joyce says:
Deming learned it’s not a problem of the people it’s a problem of the system that people work within. He found that if you want to change behaviour, then you need to change the system, and change management thinking that creates it. Doing so, culture change is then free.
To change someone’s behaviour, you’ve got to change that person’s situation.
Kaizen (改善), Japanese for “improvement”, or “change for the better” refers to philosophy or practices that focus upon continuous improvement of processes.
Teams I coach frequently ask me for “best practices”. Do not assume that “best practices” in previous projects will be equally successful in another project. In some cases, “best practices” from one context can be counter-productive in another context. Practices and processes from previous projects should be used for learning and improvement. No practice or process is both complete and optimal – once we master it at one level, we see deficiencies that were previously hidden and the cycle of improvement begins again. You should always challenge yourself to experiement and find better ways of doing things and beating your own standards for excellence –
Improvement usually means doing something that we have never done before.
— Shigeo Shingo
Mistakes are a part of being human. Mistakes are not a sign of failure. Appreciate mistakes mistakes for what they are – precious life lessons that are used for learning and for others to learn form. Part of continuous improvement is learning from mistakes. The only failure is the failure of not learning from your mistakes.
Insanity: doing the same thing over and over again and expecting different results.
— Albert Einstein
James Hird, the coach of the Essendon Football Club (a team in the Australian Football League (AFL)) took over the coaching of a team this year that was struggling to win games in recent years. A priority for Hird was improvement in the team – “We’re trying to push for continual improvement.”
The club started the season well winning several games, however, they also made many mistakes along the way and had some big losses. The club learnt many valuable lessons from those demoralising losses which enabled it to obtain an all-important victory against the best team in the competition. One of the team players said:
“The coaches were never too up when we won and they were never too down when we lost.”
“The coaches are all just about learning, evolving and improving.”
Even when the team won games and making the finals was in sight, it was still about improvement and getting better. Hird is solely focused on continuing to develop his team – “improvement first, finals second.” Through continuous improvement, Hird believes they will build a good team that can compete effectively and achieve the goal of winning a finals premiership:
“Whether or not we make the finals this year… in 18 months time we want to keep coaching and teaching so we do become a good team.”
“Our team is not built on superstars it is built on all-round effort across the board.”
In a previous post, IDEO had a fail safe culture and which allowed them to fail often in order to succeed sooner:
Enlightened trial and error succeeds over the planning of the lone genius.
It is important organizations foster continuous learning so teams constantly get better at everything they do—improving their work, making decisions, holding good meetings. That’s why successful teams emphasize continuous learning, always going over what they’ve done, identifying what went well and what didn’t, and finding ways to get better the next time around.
The following from Gemba Panta Rei highlights the importance of learning from mistakes through continuous improvemment:
Here are three important lessons from Irish novelist and playwright, winner of the Nobel Prize for Literature, and wicked wit George Bernard Shaw:
“A life spent making mistakes is not only more honorable, but more useful than a life spent doing nothing.”
We often say that continuous improvement depends on the ability of people to experiment, make mistakes, reflect and learn from the mistakes. It is truly honorable when leaders create an environment in which it is safe for people to experiment, fail and learn. In fact it may take more courage to allow others to make mistakes than to make mistakes oneself. The good results of these experiments we call kaizen, and the learning that occurs from poor results we call development of human potential. When we achieve good results free of mistakes we call it competence, but mostly it is luck.
“The power of accurate observation is commonly called cynicism by those who have not got it.”
While opinions and ideologies can be had and held with only minimal effort from our armchairs, facts require that we put forth some mental and physical effort. The act of observation requires presence and attention, the accuracy thereof open-mindedness and the will to release held opinions when they conflict with observed fact. Seeing the truth can be uncomfortable. Having truth that conflicts with one’s belief can be even more so. Pointing out the misconceptions in the minds of others can result in being called cynical or worse. Accurate observation must be preceded by and paired with education, alignment and commitment to act to improve a situation, regardless of our preconceived notions in order for continuous improvement to take root.
“The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man.”
I am an Agile Plumber – I eliminate waste, remove blockages and increase flow.
This week we had an agile workshop to establish some goals and create some epics for next quarter’s release for the agile change team at an organization I am coaching.
One of the items we covered in the workshop was to come up with a new team name for our agile change team as a result of some organization changes. Some of the names included:
- Agile Change Team (A.C.T.)
- Agile Capability Delivery Centre (AC-DC)
- The Enablers
- Centre for Agile Practices
….and there were many more.
When we went to vote on a name, a clear winner was Agile Plumbers. This name was quite humorous at first and we all got a good laugh, but the name is symbolic and there is some serious intent behind it.
An important aspect of agile and lean is about eliminating waste, removing blockages, and increasing flow which can really can change the way you build software.
There are many forms of waste including partially done work, extra processes, extra features, task switching, waiting, motion and defects. By looking at the waste that is in the steps you are doing and removing the waste you can do more with less.
We also have failure modes because we’re not paying attention to what it really takes to be agile. There are many roadblocks and blockages to getting to success. There are roadblocks that occur within teams and projects, but there are larger organizational roadblocks which are impediments to any agile approach. Agile touches on many aspects of an enterprise organization and changes many existing organizational processes that are more acquainted to waterfall. Examples include:
- Procurement, such as selecting and working with vendors.
- HR, especially what type of people we want to recruit, and changing manager’s KPIs to align more with a collaborative culture.
- Finance, how we structure the organization’s funding model to align with agile approaches and incremental delivery. Also funding may need to change to support a lean pipeline of work (continuous flow of work), rather than ‘individual projects’.
- Legal, how we write contracts that support agile teams and approaches.
- Training, creating training programs for agile (not only including agile specific topics, but also leadership and soft skills that are required in a collaborative organization).
- Governance, most traditional governance does not work with agile.
- Add your own organizational impact and blockage here that needs removing.
Finally, a goal of lean and kanban is increasing the efficient delivery of value through limiting the work in progress (WIP), making work visible and increasing the overall flow. Optimizing the entire value stream, from initial concept to consumption is fundamental to increasing flow of value through the organization. The idea is to get an idea into the development pipeline and out to the customer as fast as possible by looking at the existing processes and improving them and in the process eliminate waste and create value.
The other advantage of being called an Agile Plumber is that the name creates an memorable impression. In a large organization with many business units and organizational teams, one thing that can be difficult is being able to identify where agile help can be found.
So next time you need someone to help eliminate waste, remove blockages and increase flow, call your nearest Agile Plumber!
This is all too funny! Whilst the video is intended to be humorous, the pain of the “Agile Guy” maybe all too familiar for some. Agile changes many things we have become use to over many years – agile questions the status quo and challenges our muscle memory……..
I saw a few of these signs posted up around an office site I visited today and was amused by it. The office cubicles were also very high (‘Dilbert’ cubicles come to mind). In fact they were high enough that we couldn’t see people’s heads. The workspace was definitely reflective of a non-collaborative culture.
If you go to an agile workspace you will generally find it a hive of activity with a lots of collaboration and verbal communication. Anyone not familiar with how agile teams work would mistake this as (unproductive) ‘noise’ and wonder how people get any work done. However, most people in agile teams get use to the ‘noise’ very quickly and are able to filter out the ‘noise’. The high amount of ‘noise’ on agile teams never bothered me. I found myself naturally developing selective hearing whereby I only tuned into conversations that were relevant to me and ignoring the rest. That’s why co-location is important on agile projects – to participate in the ad-hoc conversations as well as having the richness of face-to-face communication.
I have been asked countless times “how is documentation managed on agile projects?” or “can you give me a template for agile documentation”. I have also heard statements like “we do not need to do any documentation when using agile”. And more recently I was asked “how do I best document an agile build as it progresses….I want to build up a picture of what it does by documenting the requirements”.
To start off, let’s get one thing straight – documentation does actually exist under agile, albeit not in a traditional sense. One of the most common misunderstandings is that agile = no documentation. Agile is not anti-documentation or agile does not disregard or exclude documentation. If you look at the Agile Manifesto, it states Working software over comprehensive documentation. Now where does it state that there is no documentation? The intent of this statement is that whilst documentation is important, the real value is producing working software that does something valuable for the business.
Anybody who actually works with code or close to the coalface knows that (overly) detailed and comprehensive documentation ages very quickly and actually becomes an impediment to respond quickly to business change as it diverges from the source of truth (the code).
One of the Agile Manifesto Principles that sums up the approach to documentation very succinctly and that is simplicity the art of maximizing the amount of work not done is essential. Let’s be pragmatic on what documents are really needed and keep it simple. Simple in number of documents needed, simple in the amount of detail and simple in the formality. Like most agile approaches, we produce just enough documentation. Keep documentation lightweight and simple – wikis are great and photos taken with a (phone) camera of whiteboard sketches often suffice.
The question you should ask yourself is what “value” does the document add to the process? What purpose do the documents serve? Who is the audience?
Documentation to aid collaboration has a different purpose to documentation that is needed for prosperity purposes. Thinking about documentation in these two types helps decides how we go about it. Documentation for collaboration should be kept as light wight as possible (and can be a thrown away when done) as we prefer people collaborating with each other face to face. Treating documentation for collaboration as if it was for prosperity often slows us down as the amount of detail and the content is quite different.
- Story cards
- Story walls
- Flip charts
- Agile Management Tools
- The code
- The tests
- Recorded demos
- (Valuable) documents
There needs to be a pragmatic balance of documentation (requirements, architecture, design) along with well structured, clean and readable code.
Clean code can be read, and enhanced by a developer other than its original author. It has unit and acceptance tests. It has meaningful names. It provides one way rather than many ways for doing one thing. It has minimal dependencies, which are explicitly defined, and provides a clear and minimal API. Code should be literate since depending on the language, not all necessary information can be expressed clearly in code alone.
— Dave Thomas, from Clean Code by Robert Martin (“Uncle Bob”)
This is an example where disciplined agile engineering practices is vitally important to the agile process. Don’t be afraid to rely on team member to team member communication to distribute knowledge about the system. Use pair-programming to share understanding with others so there are no dark areas and less need for people to document what has happened.
Another recent question I was asked is “how is business modeling handled in agile?” The Agile Modeling website provides some guidance. By Agile Modeling I don’t mean just UML – think ‘just barely good enough sketches’ on whiteboards – everyone should be comfortable bringing them on and in the language of the domain (ubiquitous language) – see Eric Evans’ Domain Driven Design. Domain Driven Design provides some guidance on how you think of the domain, the language you use to talk about it, and how you organize your software to reflect your improving understanding of it.
Automation of tests is important at both the whitebox (unit tests) and blackbox (functional) levels. These automated tests describe the behaviour of the system at the code level and functional level and as such by nature are a form of documentation themselves. For example, a glance through a suite of functional tests, you should be able to get an understanding of the functionality of the system. Unlike documentation, there is no fear of automated tests getting out of date as these will be constantly maintained to reflect the behavior of the system. Therefore, clean code and automated tests are seen as the primary repository of detail on how the software was implemented.
There is no one right answer as to what documentation, how to write it, and who should create it. However, constantly ask yourself before producing a document whether it is worth the effort to create and maintain it. Keep documentation simple and produce just enough. Documentation doesn’t have to be perfect – it just needs to be good enough for the purpose they serve. Once there is no use for it, stop producing it.
“Enlightened trial and error succeeds over the planning of the lone genius”
is a quote from a great video demonstrating how teamwork and agile concepts at IDEO is used to successfully deliver innovation and solutions in a non-IT environment in just 5 days!
Some highlights which I thought were pertinent to agile:
- The people are not experts at any given area. They innovate by using and applying the process.
- The person leading the team is the leader because he is good with groups, not because of seniority.
- There are no titles, no permanent (role) assignments – everyone is a generalist but brings specific skills.
- Everyone communicates and share what they learn.
- Iterative development with showcases/demonstrations of prototypes to get feedback for the next iteration of the product.
- Some of IDEO’s mantra for innovation:
- One conversation at a time
- Stay focused on topic
- Encourage wild ideas
- Defer judgment
- Build on the ideas of others
- No criticisms (this is self moderated)
- Time-boxing, to prevent the process from going on for ever (Parkinson’s Law and Pareto Principle comes to mind).
- Team judges what are the best ideas (team votes).
- Fail often in order to succeed sooner (fail safe culture).
- Having an open mind – think outside the box.
- Belief that focused chaos can be constructive.
- High use of visualization tools, story boards, information radiators.
- Fresh ideas come faster in a fun place.
- High level of collaboration.
- Teamwork, teamwork, and more teamwork!
Agile welcomes changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
Short feedback loops in Agile allow the team to get fast feedback and incorporate changes required by the customer and reduce the possibility of the team developing the wrong product for too long.
The Product Backlog is dynamic and evolves over the lifetime of the product and requires constant grooming and refinement. The Product Backlog is continuously updated by the Product Owner to reflect changes in the needs of the customer, new ideas or insights, moves by the competition, technical hurdles that appear, and so forth.
During an iteration, the Agile Team should invest some time to work with the Product Owner to refine the Product Backlog (a.k.a. Product Backlog Grooming). This includes requirements analysis, splitting large items into smaller ones, estimation of new items, and re-estimation of existing items. Priorities can change and requirements can move up and down in the Product Backlog. When new items are discovered, they are inserted into the Product Backlog causing the lower ones to move down the list (and possibly fall outside of the scope of the release) or old requirements removed as appropriate.
Although Agile welcomes changes, when using time-boxed delivery like Scrum, there are no changes allowed during the iteration, and the end date of the iteration does not change. In return for not making changes during the current iteration, the Product Owner can make any changes they want to the Product Backlog before the start of the *next* iteration. The Product Owner can add, remove, reorder, or change items before the start of the next iteration. This enables the Agile team to make and keep commitments, it gives the team focus and stability during the iteration, and it trains the Product Owner to clearly think through what is in the Product Backlog. Due to the short time-boxes, the team should be able to get some reasonable certainty into the planning and requirements and avoid churning during an iteration.
Overtime if you find that the Product Owner keeps needing to change requirements mid-iteration, then perhaps the iteration length is not short enough. Plan iteration durations around how long you can commit to keeping change out of the iteration.
How many times have you heard, “that’s what I said, but not what I meant or wanted”? Too many times I would suspect. Many symptoms of project issues are related to organizational, collaboration, and communication issues, not technology. One of the challenges of any I.T. initiative is getting people to communicate and collaborate together, especially between business and I.T.
Wikipedia defines it as:
Communication is the activity of conveying meaningful information. […] Communication requires that the communicating parties share an area of communicative commonality. The communication process is complete once the receiver has understood the sender.
The last sentence above is quite important and often overlooked. We rely too often on using static channels of communication, such as sending an email or word document to communicate a message. However, the receiver of the message may not have understood what your intention is and we often ask ourselves “why people don’t understand it? I sent them the email with the requirements!”.
The sixth Agile Manifesto principle reminds us:
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Getting people together face-to-face provides richness in our communication through, tone of voice, eye contact, visual gestures and the ability to convey tacit knowledge.
Collaboration is working together to achieve a goal. Agile And again, we can look to the Agile Manifesto for some guidance:
Customer collaboration over contract negotiation
Business people and developers must work together daily throughout the project
The best architectures, requirements, and designs emerge from self-organizing teams
During a team discussion, I liked what a fellow colleague said on what collaboration meant:
Collaboration means acknowledging that you don’t have all the answers and that the collective wisdom is better used to solve problems. Collaboration is being vulnerable and being transparent so the conversation brings out things that were otherwise not consciously known.
My communication and collaboration experience
Over the last several weeks I have been coaching and facilitating several inception/discovery workshops that included both business and I.T. One particular workshop was attended by 35 people from all levels of the organization with some participants traveling from interstate. The goal of the 3 day workshop was to discuss and plan how we would tackle a large program of work by identifying the program vision, goals, drivers, measures and the various streams of work that would make up the program.
In the 3 days we accomplished what previously would have taken weeks (possibly months) to complete. The face-to-face workshops provided a rapid, efficient and effective way to for people to communicate and collaborate. An important outcome was obtaining a shared vision amongst all participants. After the 3 days everyone walked away with a shared understanding and vision that they can take into the various projects that would make up the program of work.
What I also observed was the large amount of informal communication and collaboration that occurred outside the workshop hours such as during breaks and lunch that otherwise would not have occurred. Getting everyone together allowed whole team discussions which I believed lead to greater buy-in by all team members.
There was a high level of collaboration between everyone in clarifying business requirements, the problem space and each others understanding. The power of bringing people together to communicate and collaborate to achieve a goal was powerful. Trust was being built between business and I.T.. The business was able to understand the complexities and risk involved in I.T. development. And the level of negotiation between the parties allowed mutual decisions to be made and the quality of the decision making was made possible by having the right people together.
Agile stresses the importance on communication and collaboration. Alistair Cockburn recently remarked how important it is to increase collaboration between people and across the enterprise. Effective communicators realize that the goal is to share information, and that the information sharing is typically a two-way street. Through conversation we can iterate over meaning and seek clarification. The ability to answer questions in real time is important because questions provide insight into how well the information is being understood by the listener. Face to face collaboration also reduces the cycle time of information sharing through continuous flow of tacit knowledge.
Agile is about bringing people together to achieve more than they could individually. It is only if we have consensus through group collaboration that we can provide optimal solutions to address business problems. So next time you want to bridge the business and I.T. gap consider increasing the amount of real-time communication and collaboration between the two parties:
- Get all the players in the one place and talk.
- Engage the right players to make decisions.
- Leverage the knowledge of the team – listen as much as you talk, ask questions when you don’t understand, question if you do not agree. There is no silly question!
- Clarify and iterate over meaning to get a common understanding:
- “So what you’re saying is…”
- “Let’s use an example…”
- “Doesn’t that imply…”
- Find the value that each member adds to the group.
- Think ‘one team’. We stand or fall together. Collaboration expands our potential to succeed.
- Negotiate to reach an understanding, resolve a point of difference, or to produce an agreement upon courses of action.
- Document and capture important information so it is best not forgotten (don’t document just for prosperity sake) – it is the dialogue that is important.
Software development is a creative process and any creative endeavor requiring teams collaborating together to share knowledge, learning, ideas, solve problems and build consensus. Communicating and collaborating will always help you steer your way to a far more successful outcome.