A significant part of my current workload revolves around helping engineering organisations of all kinds succeed at what they set out to do. One client may require some outside perspective to improve their product engineering and delivery. Another might seek to simplify and adapt their internal processes. Some prefer a discreet guard rail while they set out on their own to experiment, while others seek a direct impulse and hands-on coaching and feedback. Many share a single but profound problem: a near-total lack of data and insight into their product.
While this could be a piece about observability (instrument your code, people!), it is not. This is a piece about making the most out of your current situation while you dig your team out of it (seriously, instrument your code. structured events beat metrics and logs any time of the day). It is a stop-gap measure to stop you from spiralling further into the dark unknown.
the reason o11y comes first is because it’s like turning the light on in the room. before you get all fancy with your chaos engineering, or your microservices, or your orchestration and meshes, whatever — wouldn’t it be cool if you could just, like, see what you’re doing?
At a very bare-bones level, chances are that if you are building a product of any kind, once in a blue moon you are going to get asked how it does. And you better not fire up your trusted SQL client with that deprecated-but-still-working access to your production database to answer that.
Surprisingly often I spend some of my first days with a client listening to a product owner, engineering manager or whatever title you may encounter in your day-to-day, wiping a tear from their sad, soulful eye and wailing*
I’m flying this plane with a blindfold and a hand tied behind my back. Yesterday we noticed our product being completely down because a customer bothered to ring and tell us it was “a bit slower than usual” and would we mind taking a look. How am I supposed to meet my management-set product/sales/whatever goals if I don’t even know how many customers actually use (or can use) our product?
* this is of course exaggerated, though not as heavily as you might think
And surprisingly often, they are factually correct. Now to be fair, some of these organisations might be to small to have invested time and/or money into this whole data-gathering-and-analysing business. And some might be so large that they invested, quite significantly, into humongous software offerings which strangle what remains of their productivity while they fight to utilise even 5% of the capability they bought (looking at you, IBM). Either way, they lack insight into how their product is actually used and how to improve on that.
However, if your core business loop generates revenue or makes use of customers’ personal data (hello user account), good news! You are already required to keep track of these, and probably did since Day One™. And all of a sudden you can see the twinkle of gold at the bottom of your personal muddy creek… now if only there was a paddle.
Get two for the price of none!
Next step: loop back and tie these numbers to specific events in your business process. Track customer account creation at business logic level (or at database level, I won’t judge if you truly are at naught currently). Instrument the code handling your source-of-truth data creation (eCommerce: checkouts, banking: transactions, insurance: policies, …). Try to have a coherent trace of a business event, starting from a customer entering your site (from where?) via what happens on your page (landing pages? search terms?) to the desired outcome.
Take a look at these as often as feasible, depending on the amount of automation available to you. And all of a sudden you can start thinking about how features you add or remove influence these numbers.
Now let me make perfectly clear that this sort of brick-and-mortar data, while probably keeping your business alive and profitable, can (should?) never be more than a stop-gap until you figure out what insights are valuable and helpful to you. By their very nature, these metrics can only tell you how many customers you have and how you are doing at making money. That is nice to know, but if you stop there I pretty much guarantee that you can watch both of those numbers dwindle as your product dies (or stops growing, which might be the same depending on how aggressive slash innovative your competitors are).
But if you have nothing, plain simple nothing, keeping track of these low-level, core business loop metrics in a structured, orderly, time-series-sort-of-way will help you gain trust in your product development and validate further development. You might even be able to run some rudimentary experiments. Data trumps opinions most of the time.
If we have data, let’s look at data. If all we have are opinions, let’s go with mine.
– Jim Barksdale
Those numbers will help you move forward (not unlike how a drunk person, at 4 in the morning, pouring out of their taxi, is moving forward) while you develop the capabilities required to actually look into your product as it’s being used. And while they likely will never be anything but crutches, they certainly help you along as you go from stumbling to walking to running your product like you should.
This will not get your plane in the air all by itself, but it will greatly lengthen the runway you have until you need to be airborne and flying.