Lean Software Management BBC Worldwide Case Study is Published

I first knew David Joyce, a Systems Thinker and Agile practitioner, through the local Limited WIP Society user group in Melbourne and in more recent times I have had an opportunity to know and work with him.  David is one of those people who has a knack of explaining something that you can readily understand and comprehend.

David along with Peter Middleton have just published an IEEE paper, Lean Software Management: BBC Worldwide Case Study that contains anecdotal evidence over a 12-month period, lead time to deliver software improved by 37%, consistency of delivery rose by 47%, and defects reported by customers fell 24%.

These are impressive and enviable results.

David’s paper shows how the use of lean methods and thinking including visual management, team-based problem solving, smaller batch sizes, and statistical process control can improve software development.

Quote from David Joyce’s website –

I believe it will be one of the most significant papers in Software Engineering this decade.

– David Anderson

Great stuff David J !!!

How People and Process Enabled Facebook to become a Phenomenal Company

I came across a case study on Facebook and how two key factors that contributed to Facebook’s success: people – in particular Mark Zuckerberg himself – and process – ‘moving fast’.

Facebook has a culture of speed and values moving fast above anything else – fast releases and fast learning through fast feedback.  Everyone is responsible for quality.  There are mandatory code reviews and there is an emphasis on coding conventions. There are automated tests at all levels of the stack, integrated with continuous integration systems and other tools.

There is a firm belief at Facebook that nothing should stand in the way of an engineer who wants to release a change or a new feature to production. There is no separate QA team. Engineers are responsible for the quality of their work. Changes are released daily, often multiple times a day. New features are released on a weekly basis.

Facebook doesn’t publicize that they have a process, however, Facebook’s culture of speed seem to have it’s roots in Lean, Scrum and XP.  Though not explicitly stated in the article, I get the impression that concepts from Kanban are used.  They have small batch sizes and measure cycle time to enable fast flexible flow from idea to development and then to release into production.  Many of the ideas come from Hackathons.  Facebook has a focus on human interactions, close collaboration and low ceremony to enable fast releases of working software.  High communication bandwidth between the team members and product managers is important.  Co-location of the team in the early days was critical from the Harvard dorm rooms to the houses they rented in Ralo Alto.

Agile software development embraces decentralized control and the benefits this brings, in particular with respect to moving fast. It should come at no surprise that Facebook is basically an agile shop, even if you won’t hear much about “Scrum at Facebook” or “XP at Facebook.” They work in small, interdisciplinary teams, and they deliver high-quality software early and often.

To ensure everyone is on the same page and not wasting effort working on the wrong thing, there is a common and shared vision at Facebook which enables the entire company to move in the same direction.  Getting a shared vision is a challenge in many large enterprises – most people don’t even know what the vision is let alone share it.  Facebook seems to have this nailed:

Mark Zuckerberg’s vision is deeply shared and understood not just by the management team, but by every single Facebook employee.

Reading the case study, I get the feeling that there is a tolerance of failure and mistakes. If Facebook fails, it learns, regroups, adapts, and develops some more – fail fast but recover quickly.

Albeit a several pages long, I encourage you to read the entire Facebook case study – it makes for an interesting read (even for not I.T. folks).

Be sure you read the comments by Don Reinertsen, Alistair Cockburn, Jeff Sutherland, Yishan Wong (former Director of Engineering at Facebook) towards the end of the article.

Related Links:

Cargo Cult Agile – It’s More Than Just Stand Ups

Most people associate daily stand ups or daily Scrums when they think of Agile.  And they are often the first practice that is adopted. I have observed some teams adopt the daily stand ups and a little of other practices (often with smells/anti-patterns) and say they are Agile.

The adoption of agile methods and practices is clearly producing enviable results by improving quality, time to market, productivity, increasing transparency and ROI. Agile is simple, but it is not easy. Few things that are truly worth doing are easy to do, and the adoption of Agile is no exception. As a result there is a propensity for Cargo Cult Agile teams looking for a silver bullet.

Cargo Cult refers to a phenomenon during World War II in the South Pacific where the natives observed what the Allied Forces were doing. The natives were not aware of the modern civilization, however, they did observed that certain actions of the military resulted in supplies of food, clothing, equipment and various other cargo being air dropped. When the war ended, the flow of goods and cargo ended too. Not understanding the underlying mechanisms for the delivery of cargo and in an attempt to attract further deliveries, the natives engaged in ritualistic practices such as building crude imitation landing strips, aircraft and radio equipment, and mimicking the behavior that they had observed of the military personnel operating them. However, no matter how well they mimicked the military actions, the cargo did not return.

In Agile we sometimes behave like Cargo Cult by imitating the superficial Agile methods and practices without having any understanding of the underlying essence. We adopt the daily stand up meeting because it is easy to do and many Agile teams use it. However, if the team is synchronized, communicating and collaborating well, the daily stand up may not be required.

What works for you, your team, organization, etc. can bring another person, team, etc. to a screeching halt. No practice will work for everyone or every team in every context. Before adopting any practice or technique, you should ask yourself does it make sense and does it align with the Agile values and principles? Does it help me deliver software more quickly to my customers? Does it help with achieving technical excellence? Will it help my team to be more effective?

You not only need to know the ‘what’ of Agile but also the ‘why’. Just because jumping jacks works for someone else, it may not work for you.

Defining Agile Is More Illusive Than Ever

There is a long discussion thread in the Agile Business Analyst LinkedIn Group on how to define agile.  In my post last year, The Illusive Definition of Agile which contained a link to a post from Jim Highsmith on this topic, it was alluded that defining agile would not be straight forward.  In the post I said “To me agile is not something you do, but something you are.  If you don’t subscribe to the agile values and principles and only follow the agile practices, then you are certainly going to miss the point of agile.”  The amount of active discussion in the discussion forum seems to indicate getting an agreement on the definition of agile would continue to be more illusive than ever.

The dictionary defines the adjective ‘agile’ as

characterized by quickness, lightness, and ease of movement; or nimble

And Wikipedia describes agile as:

Agile software development is a group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams.

While each of the agile methods is unique in its specific approach, they all share a common vision and core values.  They all fundamentally incorporate iteration and the continuous feedback that it provides to successively refine and deliver a software system.  They all involve continuous planning, continuous testing, continuous integration, and other forms of continuous evolution of both the project and the software.  They are all lightweight and nimble, especially compared to traditional waterfall-style processes, and are inherently adaptable.  As important, they all focus on empowering people to collaborate and make decisions together quickly and effectively.

No process is perfect.  Every approach to development has some potential for improvement.  Ultimately, the goal is to remove every barrier between teams and the success of projects, and fluidly adapt the approach, the process and the software as conditions change.  That is agility.

Can any set of principles really represent agile development?  After all, Agile is just an umbrella term for a variety of methods, most of which came about long before the term ‘Agile’ was coined.  The answer is yes:  Agile methods do share common values and principles as outlined in the Agile Manifesto.

In the end its the essence of agile you are after, not a specific method, practice or technique, hence why the Agile Manifesto is the best canonical description of the mindset, culture and philosophy of agile.

In my chat with Alistair Cockburn recently, he indicated when they formed the Agile Manifesto, the original signatories could only agree on the 4 Values and 12 Principles and couldn’t define a third level as they couldn’t find a common ground.

There are people who are relatively unknown but have substantial agile knowledge and ‘get’ the agile culture contributing to the discussion forum.  For example, Mark Hunt says:

Those of us who follow one of the agile practices don’t follow each style, practice, method, etc. as defined.  And the leaders who originated them categorically state you shouldn’t.  So many of us point to the Manifesto, as that really drives why we do what we do, and whether an idea we might want to try fits — if you can justify it under the ideas behind the Manifesto, try it! There is a whole shopping list of practices, techniques and approaches (daily scrums, using burndown charts, etc.). Therein lies the rub — just because you follow x number of these doesn’t categorically make you agile.  Nor does doing more of them or doing them quicker make you “more agile”.

He further adds:

The focus is on finding things that could improve their project, their team, speed up development somehow or other specific problems.

In addition Godfrey de Zilla contributes to the discussion:

The quest for definition I think distracts from us making genuine progress in improving this industry.

Whilst trying to define agile may give us something hang our hat on and say “we are Agile”, it is easy to lose sight of what we are trying to achieve.  The I.T. industry is a messy place and we need to make it better.

Agile is not binary, instead it is a continuum and you can become ‘less agile’ or ‘more agile’.  You can employ many different practices and if you don’t use one it doesn’t mean you are not agile.  Also with each practice you can have more or less of it.  Agility is not an end in itself; it is not a capability to be maximized for its own sake – being agile is a journey rather than a specific destination.  Rather than attempting to achieve total agility for its own sake, our goal is to achieve a sufficient balance of agility for optimum results in each project context.  Therefore, the optimum goal to pursue is Requisite Agility, which is a level of agility that balances the costs of attaining it with the consequences of not having it, given the specific details of the situation.  The Requisite Agility objective is not a fixed target. It is likely to vary over time, as the business environment invariably changes.  However, Requisite Agility can be reached and sustained over time through focused increments of work process change and continuous learning.

I think Jim Highsmith sums it up quite well in the following statement:

Agile is one of those illusive things that the harder you try to grasp it, the more it slips through your fingers. For me, the day we explicitly define agile, is the day it becomes sterile and lifeless.

Once you start defining agile with a concrete definition, it will become too prescriptive.  And once agile becomes too prescriptive, it will no longer be characterized by quickness, lightness, and ease of movement; or nimble – it will become ‘Watergile’.

Lean and Agile Games

Learning about Lean and Agile is often done best through some games.  Lean and Agile games provide an opportunity for teams to grasp lean and agile concepts and practices, share experience, realise better ways of working, and help facilitate your ‘aha’ or ‘Lightbulb Moment‘.

And they are fun!

Below is a short collection of Lean and Agile game resources you might want to try with your teams:

I have found watching David Joyce conducting Dr W. Edwards Deming’s Red Bead Experiment which sheds light on systems thinking and quality to be very entertaining. Here’s a video of David running the experiment at Agile Sydney.

You might also want to check out the AgileGames Google Group.

Growing up with Lego, some of my favorite games are to do with Lego.

Please leave a comment with any other Lean and Agile games you have found useful.

The Law of Raspberry Jam (Reflecting on Agile Progress) by Jim Highsmith

The following post is from Jim Highsmith reflecting on Agile adoption which I found fascinating.

In his classic, The Secrets of Consulting, Jerry Weinberg offers us his Law of Raspberry Jam, “The wider you spread it, the thinner it gets.” I thought about this recently as I’ve read blogs and articles from Agilists who are bemoaning the state of our Agile movement. They are concerned that the movement has gone awry, that people are practicing prescriptive agile, that they are not living up to the vision of the founders.

So, what did they expect!

As any movement expands from its narrow early base of practitioners, others take it in unforeseen directions—some good, some not so good. That’s just the way movements go. We can wax nostalgic about the “good old days,” can reflect on progress and try to redirect, or we can innovate and move forward. As we reflect on 10 years of Agile, I’d prefer to focus on the positive—how we’ve learned to deliver value to customers faster, how we’ve brought quality to the forefront in ways that haven’t happened before, and how we’ve improved the quality of workplaces around the globe.

Yes, it’s thinner than we would like. But thinner isn’t all bad. And there are plenty of individual companies and organizations that are thick. Jam, at least good jam, is lumpy—it’s thicker in some places than others. Let’s push forward on things like DevOps, continuous integration, Lean/Kanban, agile product management, and enterprise agility. Let’s stand on the jam lumps and create more of them.

More thick [insert your favorite jam here] please!

Chat with Alistair Cockburn about 10 years of Agile and more

Chris and Alistair CockburnThe photo here is a picture of Alistair Cockburn and me earlier this week (it was taken on my iPhone and its bit blurred due to poor lighting).   A few colleagues and I had the opportunity to have a hour chat with Alistair.

We started with one of my colleagues asked Alistair about what were the main points of contention amongst the signatories of the Agile Manifesto, that couldn’t be mutually agreed?  (e.g. promoting specific practices or techniques in one discipline over another).  This was an interesting question and the response was equally interesting.  Alistair said the creation of the 4 values was easy and all parties agreed quickly as they all had a common belief.  However, there were a lot of disagreements when forming the 12 principles as each method (Scrum, XP, Crystal, ASD, DSDM etc) had differences.  In the end, the original signatories agreed to disagree and finally settled on the 12 principles as you see them today after much debate.  Each wanted to go their own way to build their brand and increase market share.  You can see how some Agile certifications are part of building that brand (as well as making money).  As a result of the diversity of each method they couldn’t find enough common ground to go to a third level of detail of the Agile Manifesto.

Hearing Alistair talk about the creation of the Agile Manifesto was a good segue into a question I had – I asked Alistair to provide a brief summary of Snowbird 10 and his thoughts of the event.  He started off by saying holding it anywhere else but Snowbird Ski Resort, Utah where it all began, would not be the same.  Besides the original signatories, he also invited CTOs from Rally and VersionOne and Lean proponents  (e.g. Alan Shalloway, David Anderson).  The Vice President of PMI was also invited, but unfortunately couldn’t make it.

It seemed many good discussions were had at the gathering.  There was a lot of discussion around the concept of leadership in the Agile community and that the Agile Alliance needed to do more in this regard.   Another discussion was around Agile needing to exit the world of software and become more enterprise wide and across the entire organization.  It was also interesting to hear that there was a small contingent of people at the event who wanted to go and create a new Manifesto of some sort.

Other topics we talked about included how important it is to increase collaboration between people and across the enterprise and that is imperative that we have a continuous flow of value.  One point Alistair did make towards the end of our chat was that Agile was becoming too prescriptive.  I couldn’t agree more – teams and organizations I have assessed or come across have very prescriptive Agile approaches which is ironically not very Agile in my opinion.  I am finding that I have to coach teams to think for themselves and stop ‘doing’ Agile.  I might blog about this more in the future.

I would like to close and thank Alistair.  He was great to talk with, very insightful and he was very generous with his time to have a chat with us.

Here’s a few related links on Snowbird 10 that you might be interested in –

Lean and Agile – Doing more with less

Have you been in a similar situation as the above Dilbert?

Lean and Agile is about doing more with less.  Lean and Agile doesn’t focus primarily on cost reduction, but more on strategic alignment around enterprise agility – reducing waste and simplified processes which then could have an impact on cost reductions.  Furthermore, for comparable cost as traditional (waterfall) methods Lean and Agile can deliver higher quality product and a greater return on investment (ROI) by working on the most important and highest valuable features first – the business gets greater value for less.

Lean Software Development which has been popularised by Mary Poppendieck is derived from Toyota’s Production System.  Basically, lean is centered on adding value with less work.  Lean Software Development (Poppendieck) recognizes the following 7 types of waste:

  • Partially done work
  • Extra processes
  • Extra features
  • Task switching
  • Waiting
  • Motion
  • Defects

Some of these wastes can have a large impact on the amount of value you can deliver.  Essentially, by having less of the above wastes we can deliver more.   Partially done work means you have unfinished development and resources are tied up on items that provide no business value.  Business value is only delivered when features are fully completed.  Partially finished features result in hidden costs to projects.  When it’s time to release, you have to complete an unpredictable amount of work.  This destabilises release planning efforts and prevents teams from meeting commitments, often resulting in more work.  Methods like Scrum deliver small increments of fully working software that are usable and tested.  Scrum teams agree on a definition of done to ensure they are delivering features that are truly done, not only in terms of functionality but in terms of quality as well.

One big waste often found in waterfall approaches is doing extra steps and tasks that add no value to the actual process of developing working software.  Doing extra process steps (such as necessary meetings, extra paper work, manual reports) ties up resources which can be better used on value-added tasks and on activities of producing working software.

Extra features or the overproduction of features is often a result of gold plating.  Gold plating as described in Wikipedia:

Gold plating in software engineering refers to continuing to work on a project or task well past the point where the extra effort is worth the value it adds (if any).  After having met the requirements, the developer works on further enhancing the product, thinking the customer would be delighted to see additional or more polished features, rather than what was asked for or expected.  The customer might be disappointed in the results, and the extra effort by the developer might be futile.

On one of my past projects we had an Architect who wanted the team to build a framework for the system to cater for future needs.  This future need may not be realized as things inadvertently change.  The framework is a typical example of YAGNI.  There was no real requirement (technical or business) for the framework and there was a simpler solution which still satisfied the immediate business needs.  The framework added complexity which in return increased maintenance overheads to support it.  Every extra piece of code produced will mean more time is needed to test, integrate, compile and maintain and thus increasing the overall cost of ownership.  The key to avoiding gold plating is to ensure you only build the bare minimal that is necessary to deliver the greatest business value.  If you don’t need it now, don’t build it.  Keeping things simple is one of the Agile Manifesto PrinciplesSimplicity–the art of maximizing the amount of work not done–is essential.

To help avoid extra features, it is important to identify the minimal marketable features (MMFs).  Minimal marketable features is the smallest set of functionality that must be realised in order for the customer to perceive value.  It is the minimum viable product that has just those features and no more that is fully functional and allows you to ship the product to users so they can start using it.  The goal is to build the features that are the highest priority and has the most business value and avoid building unnecessary extra features that are less useful.  Agile’s approach of iterative and incremental development and focusing on delivering the most valuable business capabilities takes advantage of Pareto’s Principle which says you will get 80% of the business value from 20% of the effort.  This will allow you to deliver more value with less waste.

To complete more work with less time, you should avoid multitasking or task switching.  Every time you switch between tasks, you have to completely shift your thinking and gather your thoughts before you can get into the flow of the new task.  Multitasking can make you lose focus as it takes you a while to get back and remember where you were.  As a result there is a cost each time you change tasks or shift from one project to another.  To complete more work, try single tasking.

Waiting is the result of time delays and idle time (time usually waiting for things to happen and where no value is being added).  This can be waiting on people, analysis paralysis, delays in approvals, long planning times, delays in testing, and delays in integration and deployments.  Lean seeks to optimise the process by reducing the cycle time, which is the time from when an idea is realised to the time it is consumed by the customer.

Agile approaches such as Scrum emphasise the need for cross-functional teams where the team has all the skills it needs to develop and deliver fully working and tested software in each iteration.  The reason for this is to reduce the cost of motion and actions of people that do not add value to producing the product (such as traveling from one location to another to find an answer).  Furthermore, Scrum emphasises the need for a dedicated Product Owner who can guide the team daily from a business perspective.  All team members and customers are often co-located in a team room on Agile projects to reduce the amount of motion needed to find an answer.  Frequent face-to-face verbal communication is also important to reduce the motion of moving artifacts and documents.  Traditionally, hand-offs of documents between analysts to designers to developers to testers is fraught with waste and mis-communication.

Finally, Defects are a huge waste and takes resources away from producing business value (new business capability) to fixing defects.  Instead of fixing defects, Lean principles address this point by focusing on building quality into the process and preventing defects from occurring in the first place.  Agile teams deliver working, tested software every one to four weeks.  They find themselves making more changes to the code, and staying focused on today’s deadlines.  The development team and the code itself must become more Agile, which means it must be capable of faster change, sooner to keep up with business changes.  Agile teams have found in practice that Agile code is extensible, low-defect code with the simplest robust design that will work for the features currently implemented.  It is well-factored and well-protected by unit tests.  Among the Agile methodologies, Extreme Programming (XP) goes into the most depth concerning how developers can keep themselves and their code agile.  Such practices enable the team to do more changes and provide them with a kind of Tai Chi flexibility – a new feature, enhancement, or bug can come at the team from any angle, at any time and teams can make the change with confidence that it won’t compromise on the integrity of the system.

Agile development teams regularly reflect by inspecting and adapting their way of working through retrospectives.  This enables the team to make incremental improvements regularly and tackle changes in manageable, bite-sized pieces that can be actioned immediately.  These retrospectives are important in identifying and eliminating waste in processes which will allow teams to do more valuable work.

In recent times, Kanban systems are being used in software development to limit a team’s work-in-progress to a set capacity and to balance the demand on the team against the throughput of their delivered work.  By doing this we can achieve a sustainable pace of development and flushes out issues that impair performance and it challenges a team to focus on resolving those issues in order to maintain a steady flow of work.

The simple act of limiting work-in-progress with kanban encourages higher quality code and greater performance.  The combination of improved flow and better quality helps to shorten lead times, improve predictability and due-date performance.  [Anderson 01]

Through commitment to Lean and Agile approaches of waste elimination, simplified processes and continuous process improvement teams can deliver in a continuous flow and do more with less.  So next time your pointy-haired boss wants you to do more with less, look into how Lean and Agile can help you.

References

  • [Anderson 01] David Anderson, “Kanban: Successful Evolutionary Change for Your Technology Business”, 2010, Blue Hole Press.
ensure that they are delivering features that are truly done, not only in terms of functionality but in terms of quality as well.

A Dacade of the Agile Manifesto

Today marks 10 years of the Agile Manifesto.

I am so glad I met you.  You have transformed the way we work and go about product development.  You have made I.T. fun and put team back into teamwork.  Reflecting back, we have made a lot of progress in the past 10 years.  We started with Agile teams, but now we have progressed towards how we can create Agile organisations.

We still have a lot of challenges ahead and many people still misunderstand your values, but through collaboration we will continue to uncover better ways of developing software by doing it and helping others do it.

Happy Birthday.  I look forward to the next 10 years as we continuously improve the way we work.

[Edit: 17 Feb 2011]
Read Mike Cohn’s Blog Post – Reflections on the 10 Years Since the Agile Manifesto.
Visit 10 Years of the Agile Manifesto Website.

Agile Retrospectives are more than just asking questions

Here’s a great and short slide deck on Agile Retrospectives so you can get the most out of it.  Many are not utilising retrospectives effectively to drive team improvement. It is more than just asking what went well, what could be improved for next iteration and what will we do differently for the next iteration.

%d bloggers like this: