The slide above is from my DevOps 101 session. Sometimes people need reminding.

DevOps is about five things:

• C – ulture
• a – utomation
• l – ean
• m – etrics
• s – haring

First, and with a capital letter, comes Culture. Let’s talk about why.

Dysfunctional teams

As a consultant, the engagements I find frustrating are where a customer asks me in to do their “DevOps-ing” for them. And by “DevOps-ing” they mean automating a thing, (probably using some tool), or training people to do their own “DevOps-ing” by automating a thing (probably using some tool).

A conversation might go a bit like this.

• Customer: We need you to deliver some automation training for our Dev team so they can all be “DevOps-ists”.
• Me: What about the Ops/DBA team?
• Customer: No, they look after production. DevOps isn’t the Ops team’s concern.
• Me: OK… So what about when your dev team/“DevOps-ists” need to deploy to production.
• Customer: Well that’s not up to the dev team – the test team need to look at it first.
• Me: OK… will the test team be in the training?
• Customer: No, they have their own environment to look after and they are busy working through a backlog of work. We have a release in three months and they are busting a gut to get it out the door. DevOps isn’t their concern either. And besides, they report to a different person.
• Me: OK… So who is responsible for the end to end process? Will any directors or managers be in the training?
• Customer: Let me stop you right there, we just want to automate our deployments to our dev database and talk about branching. You know, DevOps? We think branching and automated deployment to dev will solve our problems managing multiple streams of development.

Branching and automated deployment to dev won’t solve their problems managing multiple streams of development. (At least, not in the context of the end to end pipeline.)

There is so much that is wrong with that conversation.

• We could talk about the bottlenecks and the large amount of work in progress backed up in test – and the ways that could be fixed,
• We could talk about the infrequent ‘big bang’ release in three months and the manual, error prone heroics that will probably be required to deliver it – and the ways that could be fixed,
• We could talk about the misguided approach regarding branching strategies or the shared development database – and the ways they could be fixed,
• We could talk about testing silos and the likelihood of drift and inconsistencies between different environments – and the ways they could be fixed,
• We could talk about the word “DevOps-ing” – and why it should be burned along with anyone who uses it un-ironically. (And anyone who uses the word “irony” inappropriately or puts their commas at the end of the line.),

But I’m not going to talk about any of those things. I’m not going to talk about any of the things the customer said. I’m not going to talk about any of the technical issues or the possible solutions to those problems.

I’m going to talk about something much, much more important.

I’m going to read between the lines. This organisation, let’s call them “Local Optimums R Ltd”, have all the problems above due to a common root cause: Culture.

Fix the problem, not the symptoms

Eliyahu Goldratt, in his seminal novel, The Goal, teaches us about the relationship between dependent events and statistical variations and the resulting problems associated with individual teams optimising for their own goals rather than optimising for the whole. (For reference, this was the book that inspired Gene Kim to write The Phoenix Project, which is considered by many to be the DevOps Bible.) The Goal teaches us that any organisation needs all functions to align towards a single goal and it promotes lean manufacturing and the Toyota Production System as a mechanism for this.

The problem for “Local Optimums R Ltd” is that the different functions are managed independently, reporting to different departmental heads, and each function is a silo, working in the way that suits their own functional requirements the most without a bigger vision. The result is that they are poor at collaborating, as demonstrated by my customer not considering “DevOps-ing” to be the responsibility of other functions, their being unwilling to invite testers or Ops/DBA folk to the “DevOps-ist” training and their refusal to discuss who owns the end to end development process.

This has resulted in various silos that can complete work at different paces, leaving backlogs for other teams to work through. To get around these problems the development team has started working on multiple projects simultaneously, while the testing team are working on a different version of the code entirely. If there are issues with the code in test either it will need to be fixed in situ and copied back to dev where one of the dev teams will need to context switch back to the previous version of the code and fix the issue.

This is a horrendously complicated development process and I can see why technical solutions like branching can seem like a fix for the problems. However, unfortunately this only fixes the symptom, and branching will introduce new and significant challenges of its own.

What if the work was managed differently, in a way that minimised or avoided the need to work on multiple development streams? Throughput and quality would increase dramatically and the organisation would be far more productive as a whole. This would be despite the fact that some of the functional teams would have been de-optimised.

So how can an organisation do that?

Well I could teach you all about the Toyota Production System and Lean and Optimising for the whole. But technical solutions alone won’t solve the problem. Technical solutions won’t solve the problem until your teams are empowered to collaborate with each other without going up the chain of command.

If all the developers and all the DBAs and all the testers and all the {insert role here}s who are responsible for developing or maintaining (or testing, or securing, or advertising, or selling, or using, or managing) the product in any way were able to speak freely with each other about the issues they were facing they might understand the end to end mechanism that delivers value to users much more clearly.

Once all the people are speaking they might be able to understand better why other departments work in different ways, and all the folks might better understand the inefficiencies caused by the conflicts between the different strategies being employed.

For example, let’s talk about that backlog that the test team are working through.

1. I bet the developers are frustrated by how long it takes the testers to get stuff done. The developers are so efficient they are now working on multiple features concurrently! (Creating more work for the under-resourced and overstretched test team. Congratulations. Have a biscuit.)
2. And I bet the test team are frustrated at how they keep being told to stop working on that thing and start working on that other thing because some new feature has just become more important – that really slows them down and they miss stuff because they don’t have time to test properly. And now they are getting it in the neck from those pompous devs.
3. And I bet the DBAs are annoyed that the SQL scripts are so buggy, aren’t the testers doing their jobs? Why can’t the devs write SQL? This means the DBAs have become wary of accepting the SQL scripts from the developers so they run their own checks before running the script on production. If issues are found in the script it gets sent back to the developers or the testers who get reprioritised again.
4. By the time a release is ready a year has gone by. The release is now so big and complicated with so many things included in it that it fails.

These sorts of issues are so common it’s becoming a joke. A very, very expensive joke which isn’t actually very funny because so many of the people at “Local Optimums R Ltd” are frustrated and unhappy in their work. Staff turnover and politics are high while the company is not shipping product as quickly as their competitors. They have entered a vicious circle where the functions are at war and the problems are getting worse and worse as a result.

I’m sorry, but setting up the dev team to automate deploys to dev, or enabling them to more easily work on more things at once by using source control branches is not going to help. “Local Optimums R Ltd” will not improve throughput or quality until they recognise the Cultural and silo problems and start taking steps to resolve them.

Plenty of IT people and “DevOps-ists” might shrug off Culture because it seems soft, unquantifiable and it is a problem that doesn’t align with the skillset which comes most naturally to them. However, it is the first and most important aspect of DevOps. That’s not to say automation, lean, metrics and sharing aren’t also important. Heck, even tools! Tools are good too. But all those things are less important than Culture.

Fixing Culture

Culture is about relationships. And relationships are built on trust. And you can’t build trust if you don’t know, understand and empathise with each other. On a personal level.

When was the last time you ate lunch with the folks from a different silo? When was the last time you laughed with them? When was the last time you talked about how proud they were when their daughter’s football team won the league or about the new blog they were working on? Did you offer to proof read it for them?

When was the last time you spoke freely with them about the pressures your function us under right now? When was the last time you were open with them about your problems? When did you last let them see your vulnerability? When did you ask them for help?

When did you last think about whether they would feel comfortable to show their vulnerability to you?

You cannot improve your software delivery problems unless your team (spanning all functions) can work effectively together. Your team cannot work effectively together unless they trust one another. You cannot genuinely trust someone unless you know them. And trust is earned, not given.

Once you have functional relationships and trust you have taken the first step to addressing your Culture problems, but without Culture – there are no “DevOps-ists”.

(Also, everyone is a “DevOps-ist”. Or, as I prefer to call them, a part of the team.)

If you would like to learn more about building effective teams, I highly recommend you read Patrick Lencioni’s The Five Dysfunctions of a Team. (Which also included in Gene Kim’s Reading List for The Phoenix Project.) Spoiler, the first of his five dysfunctions is a lack of trust.

Once you have a more functional team you will be able to adopt DevOps properly. And perhaps that will involve conversations about branching strategies and automated deployment to dev.

But then again, perhaps you’ll have some better ideas?

So to answer your question… You’re the boss. I can deliver that training for your dev team if you like. But you’ll get so much more value if you send along a few Ops folk as well.

*

While I’m on the subject – there are a limited number of early bird tickets still available for our next round of DLM Workshops. Database DevOps training for developers AND DBAs. Join a small group over GoToMeeting and build your own DevOps on a VM we host for you. Just remember to share the knowledge with your highly functional team. 🙂

Learn more below:

Jan 22^nd 2018: SQL Server provisioning and source control

Jan 23^rd 2018: SQL Server continuous integration

Jan 24^th 2018: SQL Server release management