DevOps And IT Operations

IT’s Holy Grail

DevOps is 2016’s tech holy grail – unified development and operations, both working to deliver what the business needs, quickly, reliably, and adaptably. Done well, DevOps transforms the way organisations work; it helps break down barriers between tech teams, and between technology and the rest of the business. Good DevOps is the antidote to increasing segmentation and specialisation within companies. With the promised benefits, is it any wonder that senior managers are pushing for it in organisations spanning all sizes and industries?

The reality of DevOps implementation is different. DevOps changes the way organisations run, and the way the people in them think. It redefines what’s possible and desirable. But it comes at a price. Change is hard. And some people have further to go, more changes to go through, and more to lose, than others.

The changes DevOps require in staff throughout the organisation are hard for some people to wrap their heads around. Within development, it can be difficult to think about infrastructure, security, and stability. In the business, it can be difficult to get used to the idea of working side-by-side with technical staff. For operations, the stakes are higher.

DevOps and IT Operations

There is a team within all organisations that have learned to define their value based on up-time, stability, and security; a team that have been repeatedly told that their value is in how well the systems run; a team which is invisible, until things go wrong, at which point people start shouting; the IT operations team. DevOps is a godsend for the operations team. It’s a new way for the organisation to think, where the concerns that have been given to operations, and which they have been fighting to protect for years, are shared with the rest of the company, and where out of hours support is performed by the people who wrote the system. This is a great relief. It is also terrifying.

For so long, operations teams have defined their worth by these metrics, and now they’re being taken away. What’s left? What will operations teams do, now? When developers do on-call, fight fires, and build-in stability, scalability, and security, what’s left for operations? When infrastructure is defined in code, who needs operations? Is it time to update the CV and move on, or is there a role for operations once developers have taken everything over?

People Resist Change for Valid Reasons

From where a traditional system administrator sits, it’s sometimes impossible to see what the world will look like when concerns, security, uptime, and support are shared responsibilities. This is a different experience than developers or testers moving to DevOps. Developers are being made responsible for more of the system, which enhances job security. No developer looks at DevOps and sees job loss. The end result may be collaboration, but the path that individuals in different departments take are very different. System Administrators, however, work to implement DevOps with the perception that they may be unemployed once they finish.

Having been through a number of DevOps implementations, we can say that these concerns come up regularly. DevOps is a whole new way of doing things, and requires learning how to think differently, how to use different tools, and how to work closely with people that have previously had nothing to do with operations. There’s a lot to learn, and fear makes learning much harder. If the goal is effective learning, fear must be eliminated (or at least significantly reduced).

Fear, Uncertainty, and Doubt

Much of the resistance to DevOps comes from fear and uncertainty around the outcomes, and doubt about the personal benefits. The first step in reducing this fear actually starts before implementing DevOps, or talking about introducing it. The first step is communication.

Before starting the transformation, it’s important to understand what people would spend their time doing if they didn’t spend hours firefighting, preparing for releases, telling people no, or tracking down bugs in production systems. Different people in the team will have different answers, but it’s important that those answers are clarified, and dug into, until staff see how moving past their current responsibilities lets them contribute directly to business goals, and frees up time for learning new things, giving them a compelling reason to assist in any change effort.

Once new DevOps practices start being introduced, it’s important to link individual’s motivations and the impact of implementing new ways of working together. In a well-considered implementation, business drivers, individual motivations, and new methods of behaviour will align, allowing people to work, for their own reasons, toward the same goals. This is not easy.

“If you aren’t making mistakes, you aren’t taking enough risks.” Debbie Millman

Operations staff, who have spent their careers learning how IT systems work, how they fail, how to identify problems, and how to build scalable, stable, secure systems, are invaluable to people attempting to do it for the first time. They can shorten the length of time it takes for developers to understand the infrastructure by orders of magnitude. Whether infrastructure is in code or manually installed, people who understand those systems, the impacts of bugs, and the impact of updates, are needed. And for a long time, after DevOps starts, operations will be the people with that knowledge. While they’re busy collaborating with developers and business staff to share that knowledge, they’ll also be learning from them. They’ll be picking up knowledge they’ve never had access to, before – why the business wants particular things done, how the code is going to accomplish it, and the value of the infrastructure that will enable it. Operations staff have a lot to contribute to DevOps, in the short- and long-term. They just need to see it.

Fine Print

Even the most effective transformation is going to come up against problems or obstacles. Things are going to fail. Mistakes are going to be made. More than anything, it must be clearly understood that the migration from the old way of working to the new is an evolutionary process, where there will be setbacks, and lessons learned, before the final goal is reached. And even the concept of ‘final goal’ is fraught, as it implies that DevOps is a journey with an end, with a set of artifacts that indicate DevOps success. The truth is more nebulous.

The acceptability of failure can be a difficult concept for people to accept, even when it makes perfect sense, especially if they work in operations. Which is worse, 5 hours of downtime 1/yr, or 10 minutes of downtime 1/month? In many companies, 10 minutes of downtime would be cause to reduce the number of deployments, to reduce the risk of further outages, even if it increased the likelihood of a longer outage. The truth is that more frequent outages, that last less time, are a sign of better resilience, and better systems, than rarer, more severe outages. During the learning process, mistakes will happen that increase the frequency of outages, and this is OK. Over the long run, the reduction in total downtime will more than make up for the increase in outage frequency.

It’s the Journey That Matters

DevOps is the path. It is the way of thinking, of approaching problems, of seeing problems in a unified way, that creates better results for internal staff and customers. As such, there is no final, right implementation. There are no DevOps artifacts. There are no DevOps tools. Each artifact, each tool, can be used to enable or inhibit DevOps. Each company will apply DevOps ideals in ways that suit their culture, and their customers. They’ll then reflect on what they’ve learned about themselves and their customers, and they’ll iterate. Again. And again. And again. DevOps is the journey.

I first wrote this article for the OpenCredo blog

The Risk of DevOps Tools: Automated Failure

The Promise of DevOps Tools

In the rush to embrace DevOps, many organisations seek out tools to help them achieve DevOps nirvana; the magical tools that will unify Development and Operations, stop the infighting, and ensure collaboration. This search for tools to solve problems exists in many domains, but seems particularly prevalent in IT (it may be real, or a reflection of my exposure to IT). The temptation to embrace new tools as a panacea is high, because the problems in IT seem so pervasive and persistent. I have been in many rooms, and at many conferences, where vendors are selling perfect software. No matter the problem, there’s a piece of software you can buy that solves it. I’ve heard hundreds of similar experiences over more than a decade in IT. After all that time I can safely say, there’s no silver bullet. Solving problems is hard. Understanding their causes is harder.

Automated Failure

Many consulting conversations start with a customer asking for a solution to a particular problem. Good consulting is understanding that while the customer wants this problem solved, they may not have the tools to identify the underlying cause. Sometimes a problem is pretty straight-forward (e.g. “We’re going live in 2 weeks and our system doesn’t perform as well as we need it to.”). Other times, however, there are unseen and unspoken causes that lead to the request (e.g. “Automate our deployment pipeline” is sometimes an attempt to address Development, Operations, and the Business not communicating effectively). In order for the problem to go away over the long term, the underlying problem needs to be addressed – brought into the open, explored, and resolved. This can be difficult, as some customers don’t want to talk about anything other than the superficial engagement. Other times, it can be difficult because responsibility for the problem lies, at least in part, outside the sphere of influence of the customer. It would be remiss, however, to not raise any deeper issues discovered during exploration phases, if they impact on the problem being solved.Customers ask for automation, continuous delivery, and DevOps practices. But without addressing the underlying problems, we make poor use of tools, or apply them to the wrong problems, which ultimately codifies and automates failure. Most of the challenges aren’t in poor tools, but in systemic challenges. Addressing these issues is a key component of organisational success.

Systemic issues

Deming told us that 95% of individual performance is determined by the system (e.g. Red Bead Experiment). Most of what we do, most of what the people we work with do, and most of the results, are determined by how the system has been built and maintained. And management creates the system. Both management and front-line staff are trapped by the system, but only management has the ability to change it significantly.

We perpetuate the systems in which we find ourselves. Prevailing management theory (i.e. Taylorism and its descendents) focuses on optimizing individuals – individual actors, individual teams/silos. The assumption is that by getting individuals to do their best, the entire system does its best. This is demonstrably false (see The Goal for clear demonstrations of when and how this fails).

Changemakers

DevOps, and Agile before it, attempts to change this narrative. Instead of talking about using processes to get the most from individuals, it talks about flow. Instead of units of work, it talks about customers and collaboration. It puts focus not on optimising the things each person does best, but on how information (in the form of code, much of the time) flows through the system. A good DevOps implementation will change how teams work. It will cause realignment that helps Operations and Development work together, with the result being shared responsibility and ownership of performance. This works brilliantly inside a team, or small group of teams. And it often finds a champion in management, who can help perpetuate it throughout the rest of the technical teams.

What next?

Now that information is flowing well, it’s an excellent time to introduce tools that complement the process. The best part is that with Development, Operations, and the rest of the business communicating effectively, it’s easier to see where automation and speed will help.

Each organisation is different, but tools that can be used to visualise work (physical cardwalls, Trello, Jira) are very good for reinforcing collaboration. Predictable deployments are more likely when the deployment pipeline and testing have been automated. And communication usually remains good when teams that work on the same functionality all sit together in one place.

How you fit these ideas and processes into your organisation depends on how your organisation works. Before bringing them in, make sure it functions the way you want it to.

I first wrote this article for the OpenCredo blog

Shadow IT

The Rise of Shadow IT

Almost since the release of AWS (as the first truly successful Infrastructure As A Service platform), shadow IT* has been a problem for many companies. As soon as it became easy to create new servers, needing nothing more than a credit card, development departments that had been constrained by IT Operations teams started using it. They took the ability to create new infrastructure and used it to deliver what they were requested to. In some companies, the inevitable rapidly became accepted as the way to do things, and both development and IT operations worked together to figure out how to collaborate on building systems that satisfied development’s desire for change, and operations desire for stability. Outsourcing infrastructure, and all it implied, gave rise to Devops – the unification of business needs, developer delivery, and operational capacity – but it also gave rise to something else, in companies where the operations teams weren’t quite as quick to move – Shadow IT.

Developers, long constrained in what they could request, or in the time it took to deliver on those requests, took to IAAS life it was lifesaving medicine. No more constraints! No more restrictions! Nobody telling them no! Development managers, who had been under pressure from the business to deliver no longer had to be trapped by people telling them they couldn’t. A balance which had been in place for many years was disrupted nearly overnight. Developers knew it immediately. Some IT departments still don’t know it. Any time the business tells developers to deliver something, and IT Operations tells them they can’t, shadow IT comes into play. Not because of technology, but because of people – there’s a dissonance caused by being caught between two opposing forces. When a way out presents itself, it will usually be taken. That’s human nature. Understanding that is essential to understanding that the change that has come to IT isn’t going away; it is, in fact, going to continue to change.

How do you know if you have Shadow IT?

As the CIO (or IT Director, IT Operations Manager, etc), if the business and IT are in conflict, with IT Operations saying no, then you either already have Shadow IT or will have it soon. Someone, somewhere in your development team, is considering how to use cloud technology to make deployments faster, make infrastructure more stable, and reduce their own headaches.

Why is Shadow IT a problem?

From the perspective of the business, shadow IT is both a blessing and a curse. It’s a blessing because, as mentioned above, it removes the constraints that have stopped or inhibited developers from delivering business requirements. Removing these fetters helps increase delivery speed, and capacity. The downside is twofold: 1) Shadow IT is hard to track. If IT costs belong to IT Operations (as is frequently the case), shadow IT is spend against their budget that they can’t account for. This may result in budget overruns that are difficult to explain or account for. It may also result in IT Operations purchasing equipment that never gets used, as the team is bypassed. 2) IT Operations are the part of the organisation most familiar with risks of running equipment in a live environment. They are frequently responsible for security and scalability. While some of those risks go away with cloud infrastructure (e.g. auto-scaling), not everything does, and new problems are introduced (e.g. how to handle logs for services that might disappear at any time?). These are questions that IT Operations are used to asking, and answering. While there’s nothing inherently less capable with developers, they are used to thinking of different questions, in different domains, and may not have the domain knowledge required to think of the questions themselves. For these reasons, it’s important to address Shadow IT early, and figure out how to integrate external IT systems with internal IT and business requirements.

Eliminating Shadow IT

Use of cloud technologies isn’t going away. That genie is out of the bottle. So if there’s no way to eliminate the IT, what’s the solution? Eliminate the shadow.

Since it is only going to get easier to create customised infrastructure as companies gain experience with providing what customers want, the only feasible solution is to adapt. Bring shadow IT out of the shadows. Not by forbidding it, but by embracing it. Take time to understand what it is that shadow IT provides that internal IT doesn’t. Then make a decision – is it worth spending time and energy adapting internal IT systems to provide those same service, or would that time be better spent adapting internal IT people to the new way of working. Neither is easy, but the latter, while difficult, is far more likely to succeed – people will use what’s easy and gives them what they want, after all; once they’ve settled on external IT providers, it would be very hard to get them to go back to using internal systems.

The first step in adapting IT people to using external systems is to help them understand that their role in the organisation is changing. They are no longer gatekeepers, responsible for saying now. Instead, their role is to help enable the use of the best tool for the job, bringing their specialised knowledge of systems to bear on cloud-based systems, figuring out how to adapt inherently unstable cloud infrastructure to software and customer needs for stability (hint: automated recovery and scaling). One of the biggest challenges that comes with thinking this way is that a lot of Sys Admins are used to spending their time on the command line, and creating repeatable, redundant, scalable infrastructure requires moving toward infrastructure as code (terraform, ansible, mesos, etc).

As with most new ways of working, the difficulty isn’t with the new technology itself, but in people adjusting to a new way of working that in many cases challenges their deep assumptions about what they do, and how they provide value. This change takes time, and may take new people. Don’t delay starting, however – the longer you wait to get started, the more ground there will be to make up, and the harder those same people will find it.

*Shadow IT is IT spend that isn’t sanctioned by IT Operations, and so isn’t well tracked, or admitted to.

I first wrote this article for the OpenCredo blog

DevOps Is Transformative

The Pre-Devops Environment

DevOps is transformative. This (hopefully) won’t be true forever, but it is for now. While the modern management practices of separating development and operations (and to a lesser extent, everyone else) prevail, the tearing down of the walls that separate them will remain transformative. In company after company, management and front-line staff are coming to realise that keeping functions separate, which are inherently interdependent, is a model for blame, shifted responsibility, and acrimony. It’s easy to divvy-up a company up based on function. To many people, it seems the most logical way to do it. Ops does operations, Dev does development, Marketing markets, etc. It seems much harder to do it any other way. So why do it?

Separation by function isn’t the best way to break up a company. People with similar areas of expertise work together and learn together, but they don’t understand the company, and nobody understands them. Functional separation leads to pedestals for some, and dismissal for others. In well run development and operations teams, people will tell you they’re pretty happy. They’ll also frequently tell you they wish they knew more about what was going on. This isn’t caused by poor communication, but by an inherent shortcoming in functional separation – nobody gets to feel like they own anything. Everybody is just a cog in a machine. For some people, that’s enough, but most people need more. Most people need to feel some control, some pursuit of mastery, and some purpose or sense of belonging.

Devops Is About Communication

DevOps isn’t about technology. Technology is a part of it, but it’s not the root. DevOps has been, since its inception, a process by which Development and Operations (and the rest of the organisation) can learn to speak the same language and work together. Historically, Development has been responsible for delivering change, and Operations has been responsible for maintaining stability. To many people, these are fundamentally opposed remits – change creates the risk of instability, and stability prevents change. One of the things that DevOps does is it shares out responsibility for both change and stability to both groups. Developers become more responsible for stability, and operations become more responsible for facilitating change.

The most popular implementation of DevOps is in Operations staff delivering tools that make delivery happen faster – the move toward continuous delivery. It would be a shame, however, if people thought that continuous delivery is all there is to DevOps. When some of Development’s concerns about delivery and change leak into operations, concerns with stability leak back. This is inevitable, as rapid change requires stable releases. So with continuous delivery often comes a push toward testing first, automated testing, and ensuring that both positive and negative tests are well represented in the code (this change can equally start with QA, and spread from there). Without this improvement in automated testing, continuous delivery another name for shipping bad code.

As information starts to leak in both directions, communication improves, common ground increases, and a virtuous circle is created. Improvements in communications leads to more collaboration, which can lead to changing how teams work together. In some cases, it can lead to complete automation of infrastructure deployment (i.e. infrastructure as code), configuration management (e.g. configuration as code), or at the logical extreme, developers being paged when systems go down at night (i.e. developers being responsible for stability). Improved collaboration encourages more regular communication, which in turn suggests changing team structures. If Development and Operations staff work better together than they do when they’re separated, it makes sense to put them in teams together. But how are those teams structured? It’s at this point that the rest of the company needs to get involved.

Delivering What Customers Want

Continuous delivery can only take an organisation so far. It’s part of a transformative process, part of a way to improve things, part of DevOps, but not all of it. Reaching its full potential requires assistance. It requires organisational change, which is where things get difficult. Collaboration between Development and Operations can be done by two people who want to work more closely together, two managers who want their teams to work better together, or one more senior manager who sees things and wants to improve them. Progress beyond the purely technical arena requires senior management involvement, because the next step is to look at how customer needs move through the company, and how to improve it, which usually falls outside the sphere of control for technical staff.

Customer needs usually come into the company via sales, account managers, or customer service. These are people who are intimately familiar with customer needs, and what they’re willing to pay for (as well as what they hate). From there, someone prioritises them into things to do now, later, and not at all. And then someone works on delivering, testing, deploying, and supporting those needs. These people all help needs flow through the company, and therefore help money flow into the company. These people don’t have to be separated, each delivering part of the pie, without knowing what the benefit of their work is. Instead, they can work together, collaborating and communicating on the best way to meet demand, satisfy requirements, and deliver, with a minimum of fuss. By organising teams in this way, so that customers with a need are at one end, and satisfied customers are at the other, they are transformed from groups that are looking out for their own small part of an unknowable empire, to a group that’s working together to deliver something worthwhile.

Never underestimate the transformative power of giving people a purpose they can wrap their heads and hands around, of building teams they can share that purpose with, and of tearing down walls that stop people from talking to each other. Don’t be mislead, it isn’t easy. Compensation will have to change, management will have to change, and expectations will have to change. And when it’s done, nobody will want to go back to the way it was before.

I first wrote this article for the OpenCredo blog

What to change

Understanding Change

After deciding to embrace change, what comes next? It's time to understand where the need for, and the skills, will come from. Before diving in, take some time to figure things out. While not changing anything maintains the status quo, changing without understanding the system is tampering, and is likely to make things worse, rather than better. It's important to understand the system, and it's foibles, before reacting to them. For that reason, it's worth putting aside any preconceptions and talking to the people who currently do the day-to-day work, whether it's QA, development, infrastructure, or accounting.

People's accounts of the work they do will provide evidence of which things are broken, and why. Are staff demoralised because they can't find the right libraries to do what they want? Do infrastructure and development teams point fingers at each other when something goes wrong? Are customers unhappy with timeframes for, or quality of, delivery? Do delivered features or products satisfy the letter, but not the spirit, of what was requested? These all indicate systemic challenges that people don't know how to solve; if they knew how, they'd be doing it.

Where to start

All sustainable change must include senior management. Management is responsible for the company being the way it is, chasing the things it chases, and valuing the things it values; management built the system that all staff work within. For the organisation to change, the thinking of senior management needs to change. Having seen this play out in many places, it's clear that while many changes can start at the front line, or middle management, they tend to revert over time, as the pressure of the system gradually erodes at the changes. Long-term change requires changing the way people think and, like it or not, the way people think is largely dictated by the system.

That's not to say that the front line cannot inspire management to change. Confronted by a set of symptoms from the front line (politics, poor decisions, unhappy customers, etc), or by a change process that's really effective, senior management may decide that change is required to improve the lives of customers, staff, or shareholders.

The first step in achieving sustainable change, then, is to get senior management involved. Understanding what the senior team is trying to achieve, its pressures, and its goals over the next 3-5 years, means that any desired change can be put into context of how it will assist with those goals. This is a dialogue where motivations, likelihoods, obstacles, people, processes, and technology all need to be explored, in order to ensure that things start from an agreed point. The goal is to get everything on the table, which can take hours, or days. During this dialogue, it's important to agree timelines, ensure everybody is going to be available, and designate a point person (and possibly and escalation contact) for the work. Some understanding comes about, during this meeting, of what counterproductive processes are already in place, and how they're created by management itself.

Next Steps

The next step is to meet with all staff, or as many as is feasible. The minimum required is a representative sample from everybody that works with IT, as well as everybody within IT. Most of this meeting is spent listening, drawing out motivations and goals, on the one hand, and the things getting in the way, on the other. They know, and they've either raised it and failed to achieve change, or have convinced themselves that it's pointless to raise the issues because their management will do nothing/ignore it/deny it/blow up/fire me. There doesn't need to be any evidence that any of these things will happen, only the belief that they will. Upon leaving this meeting, staff talk about feeling like they've come out of therapy, as if a weight has been lifted. Notes taken during these meetings are direct observations and quotes, which make it possible to refer back to source data whenever necessary. Direct observations will make proposed changes and inferences easier to back up with data.

From those conversations, a pattern emerges. Things stand out. Some issues occur more frequently, than others. Collate these, and do a root cause analysis - use a flowchart, or logic tools to connect the issues until it's clear which factors are causing the others. Sometimes it's processes, sometimes it's people, and sometimes it's technology. Frequently, it's a mix of all 3, as predicted by Conway's Law.

Once it's understood what's causing the problem, steps can be taken in addressing it. First, figure out what metrics, if any, can be used to determine whether a given change is effective. Most ideas about how to fix things will come from staff, though there's a chance that the changes that are needed are so radical that nobody has any experience with how to make them happen. This is where consultants are really useful. They can bring specific knowledge about how to achieve change, and how to model the behaviour you want to see.

Conclusion

Starting a change program requires a few things: understanding the system, senior management buy in, and a willingness to listen to front-line staff to see what's really happening. Without that inclusiveness, and the involvement of people at all levels, any change is temporary.

There are numerous different styles, strategies, and methods of implementing change. All we've done so far, is identify where to start it. We have yet to talk about how to achieve change, the importance of people as opposed to behaviours or actions. There's still a lot to talk about, but after coming this far, it's possible to figure out what's wrong, how change fits with the bigger picture, and where attention will be most valuable. Getting this far, despite how it feels, is the fast, relatively simple bit. Still to come? Implementation.

The Underrated Value of Listening

Unhappy Staff

You’ve implemented a change in how things work, and people aren’t happy. You spent time investigating the problem, and putting serious thought into what the issue was, and you’ve put a fix in place that you were sure people would be happy with. They aren’t. Why not?

At this point, you can do a couple of things. The first one is the one that seems to be the most common – chalk it up to ‘people dislike change’, and force things to go ahead, anyway. Eventually, people will get on board, if you’re right, and you keep pushing, and they don’t have a choice. But you’re going to have to work hard to get things embedded enough that they don’t backslide when you stop pushing. It’s a painful way to do things, and people make a lot of money being the bad guys who implement change. And when they leave, the change slowly erodes, performs badly, or gets discredited. In the long run, not much changes, and a lot of money gets spent.

The Road to Recovery

You can be more effective. Even at the point where you’ve implemented an unpopular change, things can still be recovered, though it can be intimidating to do what’s necessary. Step away from the problem, and your interpretation of why it’s happening. Put aside your judgement of the people involved. Take the time to sit down and ask them what’s wrong. People want to be heard. They want a voice. They want to know that their concerns have been understood. So take the time to go to the loudest, most upset people, and let them talk directly to you. Ask them what’s bothering them, and listen when they tell you. I don’t mean listen in the hopes of refuting their arguments, or waiting for your chance to speak. I’m talking about listening as a means of connecting to another person. Regardless of whether you think it’s justified, the people you’ve imposed change on feel aggrieved; successful change requires understanding why, and working through things with them. So, listen. Ask questions that show you’re listening (“What impact did this have on you?”, “How could things have been done better?”, etc). Don’t take anything personally. This isn’t about you, or the changes you’ve implemented, or anything you stand for or believe in. This is about them. Completely, wholly, unreservedly about them and the impact the change has had on them. Ask questions, and listen, until they no longer have anything to add. Then repeat back to them what your understanding of what they’ve said is, so you can be sure you’ve understood. This, on its own, feels cathartic to those with complaints.

Catharsis will improve things, temporarily. It will give people the feeling that someone has heard what they’ve said. The goal in going through this process is to understand where you went wrong, yes. But before that happens, you need to form a connection with the person you’re talking to. That connection is where communication really starts, and forming it comes from listening, and empathy. These, in and of themselves, are worthwhile goals. The next step is showing people they’ve been heard. This will close the loop, and make people more likely to talk to you directly, next time.

Moving Forward

Once you understand the problems that are being raised, you may or may not do anything (or be able to do anything) about it. It may be that your hand was forced, that the changes you implemented were correct, or that what’s being objected to is how things were done, rather than what was done. None of those things matter, in the moment. Stop trying to solve the problem, and just listen to what’s being said, and what isn’t. And once you’ve done it with the first unhappy person, go talk to the rest. After having those conversations, spend time thinking about what they’ve said, how to take it on board, and then communicate back to them how their advice has impacted things.

Empathy and listening are skills that are difficult for many people to master. Listening just to be listening, rather than to find a solution, is something that schools rarely teach, and work rarely rewards. It’s a skill that tends to languish, and get rusty over many years. Don’t feel bad if it doesn’t come naturally, right away. It will come, if you give it time. And the next time you want to make a change, even a fantastic change, for good reasons, talk to people first. They’ll be happy they’ve been consulted, they’ll be more likely to buy into the change that’s coming, you can learn from them, and they may even provide a solution to the problem that you hadn’t considered.

I first wrote this article for the OpenCredo blog

Making Money Through Science

Businesses exist to make money; their purpose isn’t just to generate revenue, but to create profits, now and in the future. Generating profits means delivering products or services that people want to buy. The creation of what people want is the entire purpose of delivery pipelines. (NB: The rest of this article will use ‘product’ to refer to both products and services.)

Generate an idea

Determining what people want is not easy, but it can be simple. The best way to determine whether people want to pay for something is to put it in front of them. Some companies do this by creating a completely finished product which has been fully designed, architected, and tested before a customer sees it. They want it to be perfect before showing it to customers and getting feedback. There’s only one problem with this approach: they can’t be sure what the customer wants until it’s in front of them. It’s entirely possible to waste an entire product or service delivery budget creating something nobody wants. It’s a huge risk, whose upside (if you’re exactly right) is often more than outweighed by its downside (what if it turns out the product doesn’t actually have a market?).

Hypothesise

The only time we find out if a product is valuable is when we get feedback from the customer. That means the faster we get feedback, the faster we can decide whether we want to continue, change, or cancel the product. The same is true when changing products – a change is only valuable once a customer is using it, and can provide feedback on it. (The corollary to this is that until a product is being used by a customer, it has absolutely no value.)

The logical extension of requiring customer feedback to find value is to work in ways that enable feedback as quickly and frequently as possible. Get your product in front of a customer, and see what the feedback is. Then adapt. This is also known as applying the scientific method to product development – generate an idea, form a hypothesis about the idea, create a means of testing the hypothesis, test it, draw conclusions, and adapt.

Image is CC-SA

In order to test whether your new approach works, make sure you measure what matters – how long does it take for an idea to come into the company from the customer, be developed, tested, released, and be back in front of the customers, happily being used. Don’t measure the time in each stage, as the customer can’t see them. The only thing the customer cares about is how long it takes to deliver a working, usable product, from end to end. And all you care about is whether that time is stable (with some degree of predictable variation), and whether your changes are improving it.

Test

Shortening feedback cycles means changing the way you organise product development. Rather than focus on who answers to who, and drawing it in an org chart, focus your teams on rapidly delivering products to customers. The focus should move from working on each phase of development to working on what customers need.

If you need analysis, development, and testing, then put them all together so they can figure out how to respond to customers quickly. If you write software, put in testing and deployment automation, so you can rapidly deploy small changes that customers see immediately. If you work on a large product, with lots of dependencies, start working on separating the components so development in one area doesn’t require testing and changes in a completely unrelated area.

Conclusion

You’ve changed your company. You’re now looking at how product development works, rather than on how people work on product development. And you have a measure you can use to determine success. It takes less time than it did before to deliver new features and products. They’re higher quality, and customer feedback is improving. Your measures indicate that you can get new features out the door in a predictable amount of time ( between X & Y days).

Your focus is no longer on production, but on flow. Keeping it there can be difficult, but it’ll significantly improve delivery, which will improve profits now, and in the future.

Adapt

Your first experiment worked. Now it’s time to see how you can continue to improve it. What factors contribute to product delivery timelines? After you’ve identified them, think about how changing them will change your delivery time scales, and whether a change in one area will impact another adversely. Focus on the areas that appear to be bottlenecks, and improve them until they stop being bottlenecks. Then repeat.

Feedback

Help me adapt. Leave a comment, or reach out to me on Twitter (@nrcantor) to let me know what you think and what you’d like to hear about next time.

I first wrote this article for the OpenCredo blog

Fear Kills Improvement

I originally authored this piece for The Register. This is the original, unmodified version.

Fear is the Great Motivator

Fear is a great motivator. Fear pushes adrenaline. It primes our fight-or-flight response. It primes us for a confrontation. Fear shuts down our higher cognitive functions, priming us for a visceral response, on top of which we layer rationalisation. Fear ensures continued survival of our species in times of dire straits. When we are afraid, we have only our intuition and built-in responses to draw on. When we are afraid, we cannot accept feedback, respond to things which run counter to our expectations, or learn. And so, a culture in which fear operates is one in which no learning takes place. Fear kills change.

In the last 200 years, work has changed significantly. Most of us do not face daily threats to our continued existence. But our fear response remains. We still respond with adrenaline when threatened, or embarrassed; we still shut down; we still lose our capacity for higher reasoning and openness. And it kills improvement at work. It kills creativity. And it's common.

Needless to say, fear is a response borne of perception, not reality. There doesn't need to be a tiger in the bushes in order for us to think there's one there, and react accordingly. There doesn't need to be a threat to our continued existence for us to believe there is one. In a world where we are taught to define ourselves by how successful we are at work, how much money we make, how influential we are, and how people see us, threats to our continued existence do not have to be life-or-death. Job loss, income loss, prestige loss, or promotion loss all generate a similar response. And the idea that trying something, but not succeeding at it, is frequently tied to more than one of those losses, making us risk averse, and afraid of experimentation.

"The ability to learn faster than your competitors may be the only sustainable competitive advantage." - Arie de Geus

Knowledge-heavy roles (such as IT or management) require learning in order to be effective, or help the company succeed. Learning through experience and mistakes is necessary in any role that brings about change, whether the goal is figuring out how to convert page views into sales, or transforming your business into a leaner, more effective organisation.

Knowledge economies are driven by learning - the converting of information into knowledge and understanding. Without learning, we cannot progress; without learning, our companies can't adapt; without learning, we are doomed to eventual replacement by companies with newer ideas that we couldn't test. And this is a problem, because it's culturally ingrained - we are taught not to speak out in class, if we don't know the answer; we are taught not to ask too many questions; we are taught to keep our heads down when we don't know the answer. We are taught to fear failure. And so failure gets swept under the carpet, never to be acknowledged, never to be embraced, never to be learned from.

Agile, Lean, Kanban, and DevOps all attempt to remove this fear by explicitly creating conditions where experimentation and learning are encouraged. All too often, however, the adoption of these processes fail, not due to failures in the processes, but due to their incompleteness. To see how important addressing fear is, all we have to do is look back to the theory that gave birth to every one of these movements - Deming's Theory of Profound Knowledge.

Deming, A (Very) Brief Introduction.

Edward Deming revolutionised Japanese manufacturing, starting at the end of WWII. He went into Japanese companies, spoke with Japanese executives, and told them that in order to succeed, they would have to start thinking of organisations as systems, where optimising the system and optimising parts of the system are not the same thing. Deming had 14 points for management; point 8 is

"Drive Out Fear: Encourage effective two way communication and other means to drive out fear throughout the organization so that everybody may work effectively and more productively for the company."

Deming recognised something that has been shown repeatedly through research - fear inhibits learning. Deming encouraged management to see mistakes as opportunities for learning, and failure as a stop on the path of progress.

The Value of Mistakes

Mistakes are inevitable. By punishing mistakes a company (explicitly with firing, or implicitly through reduced career or salary progression) doesn't eliminate them, it merely ensures that nobody will talk about them. And nobody will talk about the fact that nobody is talking about them. The entire organisation will collude in hiding mistakes in order to avoid feelings of embarrassment or threats (Argyris, Knowledge for Action). Any company in which failures are not openly discussed, in order to be learned from (without any sense of blame or shame), is not learning effectively.

Companies rarely explicitly punish failure. Instead, there are unspoken rules about the acceptability and impact of failure - fewer promotions, reduced raises, fewer hours, worse reviews. These unspoken rules leave no room for improvement, and must be changed for a company to start learning. Words won't change them. Changing unspoken rules requires action. Admitting failure requires vulnerability, so creating a culture that learns from failure also requires vulnerability, usually in the form of managers making their own failures, and learnings, more visible. Making it obvious that everybody makes mistakes, and only by admitting them can we learn from them. This, in turn, gives their employees the opportunity to experiment, fail, learn, and improve, more visibly.

The scope for improvement through vulnerability is downward facing, only, meaning that managers can start improving things by admitting vulnerability to their staff, but the only way to create organisational change is for senior managers to admit their failures, to themselves, to their peers, and to their subordinates.

Conclusion

Admitting failure isn't easy. It can make us feel vulnerable. The alternative is worse. The alternative is not admitting failure, not admitting mistakes, not learning from things we know are, or were, wrong. Companies which can't learn from mistakes stagnate, and eventually fail.

Fear and change are mutually exclusive. Any organisation that creates a place for one necessarily drives out the other. If we choose change, and adaptability, and learning, we choose an organisational life with difficulties, challenges, and hard work, in order to maintain a culture that embraces and reinforces learning, whatever the source. While it sounds difficult, it's the only life I'd choose.

Leave-taking and New Horizons

Over the last 2.5 years, I have worked with a number of charities, and had a fabulous time doing it. I've met some great people, with real passion for what they do, who are making a difference in the lives of the people they're helping. I was a part of that. I never delivered direct services. Instead, I helped them do better, more effective work, often through technology. It was my great pleasure.

And now it's time to move on.

I have done what I can, and found, looking back, that things were not working for me. The issues had nothing to do with our clients, or our mission. I leave behind some people who will continue to pursue working with charities with passion and drive. They will carry the torch, and continue to change the not-for-profit sector for the better.

I am moving to new opportunities, which I was too captivated by to pass up. While a lot of the work I've been doing over the last few years has been with small charities, I now have a chance to apply what I've learned to large organisations throughout London, the UK, and the world.

Many organisations have been caught out by the rate of change in technology, over the last 10 years. Traditional job roles, hierarchies, and silos, don't work when technology makes it easy to put something in front of a customer in a few minutes. The imperative to make sure everything is 100% perfect before releasing is being eroded by the idea that rapid change, and rapid recovery, is more valuable to customers. Monolithic projects taking 24 months are being replaced by very small iterations that take 12 hours. Releases which used to keep teams up late at night on a Friday, and over the weekend, are being replaced by releases which take 3 minutes and happen constantly throughout the day.

This is the future, and companies want it. They want to be able to be more responsive to their customers. They want to be able to make rapid changes. They want to be able to make changes in one system without affecting an unknown number of other systems. We will help deliver it. We help make that possible, with agile development practices, microservices, continuous integration, and configuration management, largely in cloud environments. That, alone, is pretty fascinating. And it requires changes in how companies work.

Change creates chaos. Poorly managed chaos results in failed projects, unhappy customers, and poor morale. Every change has ripples. The bigger the change, the bigger the ripples and the more likely to cause unforeseen problems. Unmanaged change has a tendency to fail, over the long term. While you can change individual processes, how the company thinks about those changes can stay the same. When the change agent leaves (a consultant, a local champion, or senior stakeholders), the old ways reassert themselves, slowly blunting (and often reversing) the changes. That's where I come in.

My new role will be to help organisations figure out what change they want to make, figure out who needs to be involved, and figure out how to achieve it. This will overlap with the technical changes that are being implemented. Working in tandem will make things more effective than either of us working alone. Together, we can change culture. Together we can change technology. Together, we can change everything.

Working at the technical level

My preferred method of engagement with a client is with the senior team. I'm hoping to bring a new viewpoint and ways of thinking to them which, in turn, will allow/facilitate changing their organisation to be more data-driven, experimental, agile, and reflective.

Sometimes it's unclear where help is required. When this happens, we get called in at the easiest level - technology - because it has discrete problems which can be captured, quantified, and solved. And truthfully, a lot of what we have to offer is about addressing technical problems. We have lots of experience solving these sorts of problems, and they can often be done with little disruption to the rest of the organisation.

Working to address things at a purely technical level requires no change. The new solution, which works better than the old one, will be fit into the organisation in the same way and same place as the old solution; if it doesn't work in the same way, it will fail, and thus we will fail. While this clearly suits some problems, it doesn't resolve the challenges that brought us in in the first place. It solves what is essentially a symptom of problems created elsewhere in the organisation (or system). Solving problems this way, without understanding the entire system, and being able to test changes made on it, is tinkering. It ultimately leads to decreased performance; the old way of doing things had known problems, while the new way has unknown problems.

Given that sometimes everybody is brought in to work at a level that's different than where they can be the most use, how do you move from where you are, to where you want to be? The key, I think, lies in framing - asking questions that allow you to expose the areas that haven't been thought of. Talk to the team you're working with, and work to address their challenges, while at the same time helping them become more aware of what's happening around them, and how those things are impacting them. The more aware they become, the more likely that they'll talk to management, and ask them to speak with you about what's going on. Lather, rinse, repeat as necessary until you're operating within your preferred sphere of influence.

This is not a quick fix. It requires deep knowledge of the team you're working with, and an understanding of why things are the way they are (what's keeping them there, and what's stopping them from trying to change), and why they want to change things (what's pushing them away from where they are, and what's tempting them to change). This knowledge gives you a thorough understanding of the current and future environments, and you can start talking about change.

The goal here is not to push change, but to be the catalyst, the agent, of change. What you bring to the table is knowledge of a better, different way to do things, and what they bring to the table is themselves, and their domain knowledge. If things go well, what you leave with is a sense of satisfaction, and what they leave with is a burning knowledge that things can improve, and an idea of how it can happen. Once that fire is lit, it's nigh on impossible for someone else to put out. With that fire lit, you become an adviser, a mentor, a guide. Instead of trying to drag people to progress, you can point it out, and let people figure out for themselves how to get there.

Learning to change

We have been working with a charity, over the last couple of years, going through clarification of their strategy, applying an IT strategy, and a few IT projects. During this time, we have also been working to help them make systematic use of data, and the organisational change that goes with making that shift. This is, as far as I'm concerned, the cornerstone of good organisational practices, and the most valuable part of approaching IT from a strategic perspective.

This charity works with volunteers who come from the UK and spend several months overseas, as part of the International Citizen Service (ICS). Volunteers apply, interview, learn what country they're going to, get any vaccinations, work on the program, and continue involvement upon their return.

Satisfaction rates were lower than expected, due to a high volume of complaints around expenses. Expenses for the program are all paid, including travel and vaccinations, which can reach £500-1000. Volunteers frequently don’t have that kind of money, so timely expense repayment is crucial.

They originally thought the an automated expenses system would fix everything, at a cost of thousand of pounds per year. They couldn't figure out why people were unhappy - they told volunteers that expenses would be repaid within 4 weeks, and they were sure everything was being repaid in 4 weeks or less (2 weeks for the repayment team to verify expenses, and a payment run every 2 weeks).

When we dug into the data, we learned there wasn't any. Nobody kept track of when expenses came in, when they were signed off, or when they were paid, so nobody knew whether the process worked. They had defined a process, and assumed it worked.

Our first step was to create a spreadsheet to keep track of when expenses came in, and when they were paid. From this data, we learned that it took anywhere between 5 and 115 days to repay expenses. Normal repayment time was anything up to 51 days.

Armed with this data, we gathered everybody responsible for any aspect of expenses repayments in a room and mapped out what the process was supposed to look like, as well as what the process actually looked like. They identified 6 things that could be changed to improve the process, some of which could be undertaken by those in the room, and some of which would require external help.

Armed with the new ideas, we talked to the senior team about the changes that needed to be made, then implemented them and measured their impact by seeing whether the expense repayment time was reduced. Once those ideas were bedded in, new ideas came up, and more after that. From 5-50 days, the team reduced the repayment time 16-22 days, from start to finish. Reliably. And continues to work on it.



Once they understood how to capture the problem, and measure it, they were able to make the process reliable, and fast. Feedback started improving. Reported experiences were better.

The cost of this project was £0, plus time spent learning, which they can now share with the rest of the organisation.

This happens frequently when digging into 'IT' problems. Things which look like IT problems turn out to be something else entirely. This isn't unusual.

So, if approaching IT strategically isn't about software or hardware, what is it about? I define IT strategy as

the application of knowledge to organisational and stakeholder needs^*, with the possibility of IT contributing to the fix, when the time is right

 No matter what needs you look at addressing it, ultimately, none of it matters if you aren't addressing end-users' needs. Focusing on stakeholder needs allows people to step outside their daily roles, seeing things across departments, and helping people understand that they're all working toward the same goals. It moves the conversation from 'accountability' to 'collaboration,' from 'What do I want?' to 'What do our stakeholders need?' With increased collaboration and understanding comes better prioritisation, and increased focus on the things that really matter.

At the end of the day, your organisation is only as effective as its ability to serve its users, whoever they are. Helping everybody focus on external user needs moves people from 'mine' to 'ours,' and helps them think about how their work contributes to the organisations purpose. Giving them the tools necessary to measure their effectiveness allows them to figure out whether what they're doing works, and provides an environment in which they can try to do things differently, and learn whether changes make things better. It becomes a short jump from this to working to constantly improve things, and when you've reached that point, you're set - it does't matter where an organisation starts; if it learns to constantly improve, it'll eventually be the best organisation of its type.

Working with Apprentices (Part II)

Having recently published Working with Apprentices (Part I), I am now saddened to be taking a new tack - What happens when apprentices leave? This post will, as ever, try to be brutally honest about where I am.

I talk a lot about how to address things so they go well. What happens when they don't? Sometimes, despite doing everything you know how to do (and learning new things, to try to help more); despite building an environment where people are comfortable; where there's no fear of failure; where saying "I don't know" is seen as an opportunity for learning, not ridicule; where coaching is a given; and where advice can be sought for any problem, things don't work out.

We recently went through this with an apprentice. Things didn't work out. Clients were happy with his work, he was curious, resourceful, and imaginative. He helped address problems in new ways, and was very popular.  We worked with him for 18 months, and spent many days coaching, mentoring, teaching, and listening. We learned about his life, his problems, his challenges, and his plans to resolve them. We approached problems as learning opportunities, and to the full extent possible, worked with him to help him figure out how to resolve the issues that impacted him and us. We acted as a sounding board and raised red flags if something was a problem.

Given all of that, how did things end up where they did?

The truth is, I'm not sure.

The details of the case don't particularly matter, other than to say that by several measures, in spite of all the good stuff, parting ways was a reasonable choice, and nobody would have been surprised if it had happened 12 months before.

But it sucks. I work for a charity that intentionally hires people who may have had difficult backgrounds. People who may not have worked before, who may have difficult home lives, who may need help, and who have not inconsiderable challenges. Not everybody who comes to us faces those challenges, but they're not uncommon in the demographics from which we recruit. Part of what we do is to hire people as apprentices, and teach them how to become people-focused IT consultants.

At what point do you say, 'In this case, we have failed. We are not the right people to provide the help that's required?' The truth is, perhaps we could help him, if we did something differently, or were persistent. Perhaps if we'd spent a bit longer, approached work a bit differently, framed things slightly differently, given him more/less autonomy/responsibility/freedom/training, we wouldn't be sitting where we are now. But there's a conflicting truth that we must face, as well. It's the same one every charity faces - time we spend working with one person is time we can't spend working with someone else. Charities that we could be working with aren't being helped, because we can't schedule meetings reliably, can't count on having someone there who we need, and can't give them the focus and attention they need.

Assuming the goal is to meet the needs of our staff and our clients, maximise our impact, minimise our harm, and help as many organisations as possible get a better understanding of IT, then we have to recognise that sometimes things don't work. We choose to draw a line, which says, 'At this point, we are no longer working toward the same goals, and we are no longer a good fit for each other.'

I can no longer help this person, but perhaps this will serve as a wakeup call, and perhaps he will finally help himself. At this point, that is my sincerest hope, and I'd love to see him again, someday, when that has come to pass. Hope, in the face of adversity, is why we do what we do. So we don't say goodbye. Instead, we say, 'Until next time,' and we hope that our paths cross again.

Working with Apprentices (Part I)

When I tell people about that we recruit people with no IT experience and teach them to be IT consultants, they frequently ask what it's like working with people without experience. After doing it for a couple of years, I can honestly say, it's no different than working with anybody else. This is frequently a surprise to people who think people without experience won't know how to behave or how to do anything. They think we'll have to be firm, or 'cruel to be kind,' to get results and to keep them from dropping off the apprenticeship.

We understand this view, but don't subscribe to it. We don't want to be the same as other companies, so we don't treat them the same as other companies. Ours is a company where we value autonomy, mastery, and purpose. We use a cardwall to ensure everything that needs to be done is visible to the whole company; break things into pieces to give us a chance to iterate and constantly improve; don't judge other's behaviour; and ensure we always focus on the area that needs our attention the most.^*

This changes things. In an organisation that keeps information hidden, people can't learn how to behave, or what to do, without making mistakes, or being told (which frequently takes the form of 'behaviour management'). By opening up the needs of the organisation so everybody can see them and can choose what to work on, we ensure that even people with no experience can find something to do that needs to get done, and which they're willing to take on. This appeals to their desire to achieve mastery, and their autonomy, and we do it for everybody.

Our process comes with risks. Sometimes people get lost. They don't know what to do, or they don't know how to structure their work so it's achievable. Our job (regardless of experience level) is to help people figure out what needs to be done, and how to make it possible. Sometimes that's me asking apprentices questions, and sometimes it's them asking me questions. It's a give and take based on equality and respect, where experience and length of service mean less than empathy and learning, and the more you know, the more likely you are to spend your time helping people who know less.

Sometimes people make mistakes. Mistakes are inevitable, if you're learning, and pushing boundaries. For us, the important thing is to learn from those mistakes - to reflect on the process that lead to the mistake, rather than who is responsible for making it - so we can do things differently in the future. We approach every project, every task, every need as the current step in an iteration. Learning is an ongoing process which never ends. Our apprentices start out shadowing more experienced consultants and working on a service desk. Over time (6-9 months), they pick up enough experience to run their own small, supervised projects. In the next 9 months, they start running projects on their own and mentoring new apprentices. Within 18 months, they go from knowing nothing about IT to running their own projects and taking responsibility for the results, challenges, and learning.

It is my sincere privilege to watch them grow and change, learn and improve, and to embrace and unleash their own potential. They learn to look within themselves for their self-worth, and I couldn't be prouder of them for it.


^*How we do these will (probably) be explored in later posts. As with everything else, this is an ongoing work in progress.

You can read my follow-up post here.