DevOps: Transforming Your Organization with the Andon Cord
Jeremy Morgan 
May 14, 2019 - 6 min read
Last Update: Sep 16, 2021
Imagine you’re working in a factory. You’re assembling Toyotas all day long, then your part won’t fit. What’s going on? You do this hundreds of times a day but now the bolts won’t go in. No reason to panic, you pull a cord to get help. Two co-workers arrive immediately. They find out you have a box of bolts with the wrong thread. They swap out the bolts, and you keep going.
At the next meeting, you discuss what happened with the team. The two boxes of bolts are the same color with a similar-looking label. It’s easy to mix these up, and that’s exactly what happened. The team puts different colored labels on the boxes so the folks stocking your bolts don’t mix them up.
Sounds like a great system right? You ran into an issue, called for help, and with more eyes on the problem you found a solution. Then you share the solution with the group.
That’s the power of the Andon cord, and it’s something you can use in your IT organization.
The Origins of the Andon Cord
The “Andon Cord” is a part of the Toyota Production System. It was created when Toyota was smashing records for production, and they did something unheard of. It was not intuitive or natural, a way to stop the production line when it was common knowledge you.. don’t stop the production line.
It was just a simple rope, like the one you pull on a bus to let the driver know your stop is coming up. People pulled it when they needed help. The ways it changed their manufacturing efficiency was astounding.
Though it sounds simple, this was a big culture change for the factory. Anyone on the floor had the power to pull the rope for any reason. They weren’t scolded or chastised for doing it. You wouldn’t get fired for pulling it too much. You just pull.
The results of this experiment? They created a learning organization.
In the words of Andrew Shafer: you’re either a learning organization or you’re losing to someone that is.
My Personal Experience with the Andon Cord
I’ve been blessed to see the entire spectrum of this philosophy. Right out of high school I worked at a lumber mill. My job was to pull boards off a moving chain and stack them. 10 footers in one pile, 14 footers in another, etc. We had big red buttons you would press to stop the chain when things went wrong. Usually, it was a row of lumber falling over, or being unable to stack the incoming boards fast enough.
If you stopped the chain, instantly things went quiet, and beacon lights all around you went off letting everyone know it’s stopped. If we stopped the chain for more than a minute, the planer and saws would have to shut down, and our former drill sergeant boss would come down the stairs screaming, asking why the chain stopped. They could fire you for doing it too many times. Though I wasn’t fired for it I remember being on the receiving end of some serious yelling for stopping the chain. I can still picture him screaming and spitting in my face yelling.
As a volunteer firefighter for our local fire district, it’s quite the opposite. If you run into a problem or large abnormality, you must pull that Andon cord and ask for help. If you come on to a scene and don’t know what to do, or have a patient in a bad situation and you need ideas, you had better squeeze that radio microphone and ask for help. People’s lives are at stake, so it’s not just encouraged it’s a mandate. It is an expectation that you stop what you’re doing and ask for help.
After the call an officer or chief will do a “blameless post mortem” and discuss why you asked for help, what you could have done, and whether the course of action was successful. We all learn from it and become better firefighters/EMTs. Learn, learn, and learn some more.
In my professional IT career I’ve seen a variety as well, but these two are the extremes. You’ll need to find your sweet spot for your organization.
Can You Use This?
So implementing this in your organization in principle is easy, at least in principle. When someone runs into a blocker, they alert the team they need help. You could do this via Slack, IM, or even email if you absolutely had to. Then the team drops what they’re doing and everyone swarms the problem until it’s solved.
You might be rolling your eyes already. You will find resistance to this idea, and the reasons are pretty easy to guess.
The Coworkers or “doers” might object because:
- Nobody wants to just drop what they’re doing for your problem.
- They’ll be too busy, or in a meeting.
- They don’t believe swarming will solve the problem.
Managers might object because:
- They don’t want a bunch of doers stopping work at once
- They may not believe swarming is effective
- Timelines may be affected by this.
These are all valid concerns. Honestly, these may be valid reasons for your organization to avoid doing it. You should at least give it a try.
The upside here is you become a learning organization, constantly experimenting, learning, and advancing your effectiveness. It’s highly unlikely you’ll pull the Andon cord and stop work for something you’ve already solved as a team, right? It’s worth trying.
How You Can Implement the Andon Cord
If you want to implement this idea with a team, here are some things you’ll want to establish before starting out.
- Identify your fixers - Identify common pathways for common problems. For instance if it’s a database issue, get Mark involved. He’s great with databases. If it’s an infrastructure issue, get Debbie involved. She knows our systems like nobody else. Identify your “clutch” people, chances are your team already knows who they are, but establish it formally.
- Prioritize your problems - Categorize the severity of the issue and determine a level that requires stopping everything. If it’s slow performance or a small configuration issue, you can put it under the threshold for stopping the team. If it’s a major blocker that prevents progress, bump it up in priority so everyone knows they’re stopping for a good reason.
- Store your findings immediately - The first time you “pull the Andon cord” document your findings and the solution right away. Don’t wait for it, and show the team the results. This helps you greatly with buy-in of the idea when they can see the usefulness.
Conclusion
Nobody has to tell you that technology moves fast. If you and your team want to keep up you need an advantage. By creating a culture that fosters experimentation and constant learning you’ll be leaps ahead in the battle to keep up. The days of “we’ve always done it this way” are over. This is one tool you can use to make your team more solid an effective. Try it out and let me know what you think. You can always yell at me on Twitter or LinkedIn and start a thread. I’d love to hear your experiences and feedback.
What is your DevOps IQ?
My Devops Skill IQ is 232. Can you beat it? Take the test now to find out your Devops IQ score!!