G'day everyone, Summer Alicia here with you again. Thanks for coming back. I wanted to tell you a story today about a recent uh example of not using models correctly. And the star of the show is yours truly. And it's important to show our own experiences, moreover, our own failures, our own mistakes, our own, oh, wish I hadn't done that. Because in our careers, we've all experienced the wonderful feeling of that creeping, kind of warm, cold, prickly feeling at the back of the neck when you realize you just messed up in a pretty significant way. And I'm not too proud to say I've had that a few times in my career. Um, yeah, we move on, we move past, and we grow, we learn. But I had one the other day that I want to share with you that reminded me of a couple of things. It reminded me of the concept of feedback loops and optionality. And I want to dive into those, but I wanted to give you some context first. So I got up in the morning and was checking my emails and I had a bill. And I had a bill from a service provider. You can all hear where this one's going, can't you? I got bill shock. Bill shock for the first time in a very, very long time. Mr. Tell people to monitor their usage, Mr. Feedback Loops, Mr. Put your budgets in place. Didn't. Well, there's a reason why I didn't, because all good intentions, all things come from good intentions, but without mechanisms I fail. There's another big lesson to have. But here's the mechanism that was missing for me. So obviously, doing my own thing these days, I operate under a different set of criteria compared to when I was at AWS. At AWS, I had our own accounting systems and monitoring and what have you. Don't have that anymore, which is fine. Nothing to do with them. It's to do with me. So I had built a system, I've got many uh software projects on the go, as I'm sure most of you do. I was had built a system that involves processing documents. And one of the things it needed to do was to produce these documents accurately and quickly, and so I'd wired up a call to, in this case, it was Text Tract, to pull the data out. It was pretty low volume type stuff. Um so my mental arithmetic was like, well, this is probably $20 to $50 a month type cost, fitted well within the budget I had for the application, made good sense from a business standpoint, no problem. Even if it went to $100, yeah, probably okay with that as well. So good, didn't give it another thought, charged on with my work. Back to the bill shock. Um opened up my bill the other day, uh, it was well over two and a half thousand dollars. And I can tell you when you're looking for $20 to $50 to $100, uh, two and a half thousand dollars definitely qualifies as bill shock. And so after the dust had settled, me going, what the actual um doing my due diligence to make sure what was it, did I really process that much? What happened? And what had happened is I'd taken a design decisionslash approach to the thing I was building, where I had, you can guess, increased the number of pages I was processing significantly. Obviously, the bill tells you how significantly. Uh, this wasn't a coding error, it wasn't a mistake, it wasn't unintentional. But what I had neglected to do, because I'm out of the habit of it, and clearly the habit is getting rebuilt now, is I had not set my budget alerts. So I was happily consuming without any feedback loop, without understanding the ramifications of my architectural decision. And one of the things I love most about cloud computing and developing in the cloud is that you get very fast feedback in terms of your architectural decisions and you can make the changes you need to make to fix the problems. So, what did I do? Firstly, I let myself feel bad. It's important to feel bad. And when the family was looking at me going, what's what's wrong, Dad? I said, Well, let me explain to you what happened and why I feel bad about it. But more importantly, let's use this as a family teaching moment as to what you can do about this situation. So, what was the first thing I did? Well, the first thing I did was put a budget in place. So I would have a correct budget alert across the infrastructure because that was reminding me I had zero visibility. And again, I bang on like the best of them about OODA loops and visibility and situational awareness, et cetera. I had no situational awareness of what my choice from my architectural standpoint meant. And I've seen this countless times in all kinds of projects, uh enterprise projects, etc. The person developing the system has no genuine understanding of the cost of the operations of the system or the designs they're making, not through their own fault, I'd like to say, but just because they are not presented with that information and that has an effect. So budgets are in place, but I still didn't think that $2,500 was the right number for the particular thing I was building. And this is where optionality starts to come in. And this is the great part, again, of accepting feedback, interpreting what it means, and using that signal for yourself. So I thought, self, is this the right solution for the problem I'm using? It was the right solution at the time I made the decision. So again, I made a decision based upon a set of criteria, i.e., workload and speed to market. And at the time I made those decisions, they were made with the best available information. Now, demonstrably, the bill shows me, I was wrong. Now, in many organizations, there's a bit of a blame culture. Well, you overspent, you did too much, bad, bad you. The reality is, is I made the best decision I could make at the time with the information I had at hand. And that's all we can judge. Now, what I had done wrong, and again, people aren't wrong, systems are wrong. I hadn't put in place a system to measure whether my assumption of workload was correct and whether I was tracking to what I thought was there. I'd gone quick, I was moving fast, I like to move fast, but I hadn't put in place the guardrail that would protect me against skating off the track and skate off I did. So the next thing to do was to use this again as a prompting opportunity. When something hurts, when you hit an edge, that is your opportunity to test optionality, to test fitness for purpose, to test have the statistics change, have the conditions change, etc. One of my favorite quotes is I think it was John Maynard Keynes who said, When the facts change, I change my mind. Protel, what do you do? So I'm a big fan of changing my mind when the facts change. Not capriciously, but the the data has changed. We have a different situation. And so I went on a journey, a short journey that AI makes really easy these days, to be honest, which was I needed alternatives. I needed to assess could I get the same level of quality through either a different online service? In my case, some form of local processing was also an option, uh, some sort of self-hosting was potentially an option as well. Um, it was all on the table. And I was able to specify a really robust spec that allowed me to both explore different options, but also assess the efficacy of those options. But not spending that amount of money. And in fact, if you think about the levers you have to play with, you can play with better, faster, cheaper, choose two. It's the iron triangle. Now, I'll hasten to add, the iron triangle does not apply when you're doing a generational change in technology. In fact, you can typically achieve all three. It's pretty amazing when it happens. But let's not dive into that. That's probably another topic. But I knew I could have it better, faster, or cheaper. And I had to sacrifice something in my assessment to get what I wanted the most, which was cheaper. So my trade-off for cheaper was faster. So, what I meant by that is when I set the task of the spike for the AI to do this assessment, et cetera, I said the quality of output has to be as good, if not better, than what I currently have. So that's the better. It must be essentially low or no cost. That's the cheaper. However, I don't care what the wall clock time is. So I sacrificed faster. And I actually went through two different options. I went through a whole iteration of one option that failed the test. It couldn't do it better. And it was interesting. The AI thought it was going to be better because you know you can ask AI, hey, give me some options, tell me some things, do some blah, blah, blah, blah. That's great. But a wise man once said, trust but verify. So you need to trust but verify everything AI has to say. And it had given me some advice. And the advice on the face of it, when I looked at it, thought, yeah, that's this could work. I'm not familiar with all the libraries in this space. This one seems like a good fit. Go off, do your stuff, build a harness, do the assessment, or run some tests and come back, and did that, and it was absolutely terrible. It was the wrong solution. So back to the drawing board. Let's test again. What do we need to do? What do we have to get up and running? And found something very suitable and ran it up and ran the tests. And who would have thunk it? Same or better quality, low cost, definitely takes a little bit longer to process. And would it scale for enterprise use? No. Do I need it to scale for enterprise use in my particular use case? No. Or not yet. So we'll get to optionality in a minute. So it was acceptable. The nice thing is that when you design systems and if you're designing them with some form of intent or some sort of understanding that things will change over time, you will start to habitually build in abstractions at each and every layer. And again, this is not a discussion about monolith versus microservice. This is purely a discussion of separation of concerns in either case. That's actually what's important here. So in my case, because I'd separated the concern of the transcription of information compared to the chunking of information, compared to the processing information, et cetera, it was actually quite a surgical replacement I was able to do. And in fact, none of the upstream or downstream process was affected at all. It was just that middle component, which meant that I could make the change with a high level of assurance that it was going to work. And because I'd done my testing on the accuracy, I wasn't concerned about what happens if I replace that particular component. And again, it may sound obvious, but if you've looked at any large system, you'll obviously see a lot of spaghetti and a lot of dependencies that have been taken where they shouldn't have been taken without hard firm boundaries. And this was a great example where having those boundaries is super, super effective because I was essentially able to make this change whilst I was doing other work. Because probably like most folks these days, if I don't have some form of AI agent out there doing something for and on my behalf at all times while I'm doing other things, I feel like I'm wasting time and wasting available tokens. So I was able to have this system up and running and doing its stuff and replacing itself very easily. And so this concept of optionality of choice is not so much a case of I had to abstract away all the benefits and all the usefulness to make it possible for me to make a change. It was just that I had to reasonably separate the concerns so I could make the change down the track if I needed to. And this is a very different level of thinking to a lowest common denominator abstraction. I've seen many people waste many millions and many years of time trying to have the one ring to rule them all, to abstract their infrastructure, to abstract their coding, to abstract the services they use, etc., in a vain attempt to provide their ability to execute optionality by drag and drop, or I just move from this provider to that provider, et cetera, and everything's good. There be dragons in that area. It feels like the right thing to do in many cases, it is absolutely not the right thing to do. In fact, I would venture to say it's almost never the right thing to do. So, what is the right thing to do? The right thing to do is to understand the cost of change, understand the level of dependency you're taking on something, and to understand the optionality available to you. Now, again, in today's world, it's a lot different than it was 10 years ago, definitely a lot different than 20 years ago. Typically, you have a cornucopia of options of any particular function, capability, service, etc. Closed sourced, open sourced, as a service, self-hosted, build it yourself. Like you can do whatever you want. You get to choose. You get to decide. Big theory of this uh this podcast is you get to decide. And so having an understanding in your system design and your architecture as to what are the swappable components and how long it would take to make those changes is what gives you that optionality, is what gives you that migratable ability. So I was able to move off a complete service very simply. I can use other services that exist on that particular platform. I could choose to use none. I just, you know, I don't have to I don't have to do a, you know, we're burning the boats everyone out type approach. I can do a gradual approach, I can do whatever suits me. And what that means is that my cost of change is not built in up front through some form of abstraction tax or again, lowest common denominator tax. I pay the price when the thing happens rather than trying to anticipate that the thing could happen in one time in the future if everything goes wrong. You know, because when you do these mental calculations, you can go as far as you like because everything is possible. But when you say what is probable, and then when you put in place steps that when things actually happen, you can action them quickly, easily, and reasonably non-disruptively, you have huge capability. So, really what I'm saying here is there's this combination of the feedback loop and who is getting the feedback loop, but then the ability to exercise change based upon that feedback loop. So, have I given myself a choice? Have I left doors open? You'll often hear me talk about one-way door and two-way door decisions. This is kind of an example of a two-way door decision. Not all two-way door decisions are super, super, super easy, but they're still two-way door decisions. Like, yes, I had to make a code change. Yes, I had to do some work. Yes, it probably set me back a day in terms of what I was doing. But that was worthwhile because I didn't want to pay the drawing half grand every month. And actually, I think that was only half a month's use, so it's probably gonna be five grand a month. So I totally made sense for me to make that change. Now, it could have also been that that price was fine based upon the work that I was doing, or the cost of change, or what have you, or I couldn't get something better, et cetera. So again, doing nothing is also an option. It just is a case of assessing it properly. Now I want to give you another example. This is a somewhat of a social experiment I've been running for at least the last 10 years. So I contract my mobile phone with a mobile phone provider. Uh, every two years I get to get a new phone because why not? I remember the days where getting a new phone meant you got lots of new functionality, not just something with a bigger lump on it. But you know, that's how it works. But, you know, when I get this new phone every two years, my contract expires. I need to choose what size I want, any changes to the plan, all that good stuff that you do. And I could probably do it online. In fact, I think I tried to do it online last time, but sort of gave up halfway. But anyway, that's not the point. The point is that this gives me an opportunity to experience the lack of a feedback loop. So I go into the store of this particular provider every two years, and I sit with their poor retail agent who needs to take me through what you would think is the simplest of customer transactions. So I am a post-paid customer, so they have all my credit information on file. They can tell that I've paid on time every time for at least the last 10 years, minimum, probably longer than that now that I think about it. They know that every two years I come and get a new phone. They know that I never claim anything in between, and I typically go for the second highest or highest plan. It should be a transaction, I would argue, that should take no more than five minutes. I'd even give them 10. It is, without exception, at least a 30-minute process. It's a 30-minute process of checking one system to see if we have stock, checking another system to see how you can get it, checking the previous system where you had your customer information, then another system, then the plan, then the plan won't go with the phone, then I need to recheck your credits, etc. etc. It is a diabolical disaster. And I watch these poor retail staff who are getting paid very little to do this, go through this absolute palava of alt tabbing and shifting between very crap tacular screens and trying to make things work. And I'll engage them in conversation because hey, it's half an hour. What the hell else am I going to do? And we talk about the fact that, yeah, the system is getting slower, or the system is always slow on a Tuesday, or it's been really bad since this new phone came out, or they deployed a new system two years ago. It's just as bad as the old system. I'm convinced that no executives of this phone provider ever go into a store and get a phone. You know, that secret shopper type experience, I think they would be stunned at how poorly their significant IT investments actually execute when it comes to customer experience. And one of the questions I always ask when I'm working with executives and doing briefings or what have you, is I'll ask them the question is I'll say, when was the last time you physically went out to experience the cold face of your given service, be it as a customer or a service recipient or a user, um, and actually felt what it was like and experienced the reality of it. And it's almost without exception one of the most uncomfortable moments of the meeting. Lots of shifting around, lots of harumphing, lots of, well, I you know, I'm so busy, I don't have time, etc. They have given away any semblance of a feedback loop. And so they're just not getting the signal. And so something that good executives do, and something I used to have when I was running running big teams, and I failed in my latest example when I was talking about the system I was building. So again, the Maya Corporate that I know better is building good feedback loops, both formal and informal. And not just getting the typical data you get from a business standpoint, but also getting the anecdotes and finding out what's going on. Because when the anecdotes and the data don't match, that's your trigger as a leader to dive deep. And so in the telco example, the telco may be one of the biggest telcos, it may be bring on more customers, it may be fantastic. That's the data, maybe. I don't know what the data says. But the reality ain't that. It's a painful, slow, inefficient, ridiculous process when you go into a store with no recognition of an existing customer and no ability to delight them such that they go, wow, this was so easy, I should bring my whole family on board because it would be the click of a button to do that type stuff. So again, the lack of feedback loops in your environment becomes critical. And so I challenge you, if you're listening to this, to think about A, where you can put your feedback loops in place for the work that you do. But B, if you're working particularly in a large organization, helping the leaders of that organization experience the reality of things and to get a sense of what the result of the decisions they're making are, versus just thinking they've done the right thing justifiably, but getting no alternative signal back. You know, we kind of all, I think, from time to time, need to feel the creeping warmness at the back of the neck to just go, yep, I need to reassess what's going on here. Things aren't going particularly well, or maybe not as well as my genius plan had come up with. I need to adjust and I need to change. That's a healthy thing, it's a good thing, it's a scary thing, but it's part of what you have to do. So if concepts like that are interesting to you, you can drop into the show notes. You'll see I have a set of training courses available now that you can click on and do online if you'd like. These can help you with the way you think about problem domains, the measures you apply to the work that you're doing, et cetera. It's accessible, it's understandable, it's clear, and I think it's very useful. So if you're interested in that, please click on that link. There's also another link, and it's a more of a fun link. It's a link for some merchandise that I've created, some custom shirts and the like, that I think capture some of the zeitgeists of the time from a geek humor perspective. So feel free to visit that. But more importantly, tell others that the podcast is out here and that we're having some very interesting conversations. In the next episode, you're going to hear again more guests. Guests. Again, typically it's one-to-one individual guest type episode. And I have some amazing guests with some really interesting stories coming up that hopefully will help you learn a bit about what goes on behind some of the decisions that we make and things that work, things that don't, and how to try and find the way in between. And it's always a case, ultimately, that you get to decide, and it's always going to be a qualified answer. It's always going to be yes with an if or no with a but. And figuring out where you want to be on that spectrum becomes really important. But the most important thing is that you keep on building.