I attended DataOpticon yesterday in London. It was excellent. Massive credit to Steph Locke (b|t) for organising and gracious thanks to Microsoft and all the sponsors for hosting.
Between sessions, Steph introduced me to a pair of folks who had a technical question. Steph thought I might be able to help.
This is amazing.#DataOpticon pic.twitter.com/MqHAQrUtLU
— Alex Yates (@_AlexYates_) September 4, 2019
They needed to pull data out of a source OLTP database to serve a reporting system. The data needed to be pulled out regularly, within short windows. The report was critical to the business for various reasons. However, the source database had lots of locking issues and they were getting hit with deadlocks. This meant sometimes, and increasingly often, they failed to pull out the data within the allotted timeframe. This made people sad.
My first thought was: “Uh oh, I’m out of my depth!” I’ve never claimed to be a DBA. These two were talking about all sorts of solutions and were considering various replication options. I was about to try and find someone better qualified to answer their questions.
However, first I thought I’d ask a few more questions.
“Why do you have these locking issues in the source system?”
Oh, it’s because of XYZ in the OLTP database. We know how to fix it, but we aren’t allowed to make the necessary changes.
Another team is responsible for that system and they won’t make the fixes. As far as they are concerned the database works perfectly well for their purposes and this is our problem. We don’t have permission to make the changes ourselves – so we just have to work around it.
“Okay, who is your lowest common (and effective) line manager? Surely they must understand your objectives and their objectives both serve the same company?”
Oh, crikey. There’s a question!
Would it be so and so? No, it’s probably higher up… Ultimately, they report to the COO, and we report to the CFO. So… I guess it’s the CEO?!
First time (I guess) I see a live speach to text!
This is @_AlexYates_ at #DataOpticon pic.twitter.com/0hKEHyunUt
— Alex. Lewandowski (@alexlewando) September 4, 2019
“So let me summarise…”
… they are responsible for the source OLTP database. You are responsible for a reporting process. Both your work and their work are important for your business to operate.
You have identified some problems with their database. While these problems do not significantly impact their business objective, they screw you royally.
You have proposed some fixes that would benefit both parties, and you in particular, but they are not willing to cooperate because that database is their responsibility, they are too busy, and they won’t let you make the changes yourself.
As a result, you are asking me to help you figure out a (probably) complicated work around, that neither of you really understand, and no-one really wants to maintain. You recognise this is a horrid plaster, rather than a proper fix. We all agree it would be much better to fix the underlying issue.
Not only are you unable to come to a compromise with the team directly, but you need to go right up to senior management before you find someone who has responsibility for both your objectives an theirs.
Is that a fair summary? (Apparently it was.)
I mean, I could introduce you to someone here who can probably help you to work around this issue. I’m sure they will have a great answer to the question you asked me… but I don’t think that’s really going to solve your problems in the long term. Do you?
I suggest you start asking some different questions.
10 minutes after this conversation I delivered my session. “Essential skills for data folks in the age of DevOps”. The image at the top of this post is one of the slides I used in that talk.
Sometimes it’s important to remember these things.