The Qualified Answer
The Qualified Answer dissects the real decisions senior technology leaders make - what was chosen, what was traded away, and what they wish they'd known.
Because every honest expert answer comes with conditions. This podcast unpacks them.
The Qualified Answer
Averages Lie, AI Washes, the IDE Loses Its Crown
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Send a text message to the show!
You measure your systems with averages, your board deck is full of "AI," and you barely open your IDE anymore. Three things worth a second look — because each one is a signal, and the signal matters more than the noise.
In this solo episode, Simon makes the case for measuring at the tail instead of the mean (and why the 99th percentile is usually where the gold is), unpacks why AI washing is just cloud washing wearing a new label, and examines the quiet, seismic shift of the IDE losing its place as the developer's centre of gravity. The connective thread: when a core part of a value chain is fundamentally altered, pay attention — something is happening, and it's worth figuring out what.
Three observations, one habit of mind: know where to look.
Hosted by Simon Elisha — former AWS Chief Technologist, 35+ years in enterprise technology.
Subscribe to The Qualified Answer newsletter: https://pages.thequalifiedanswer.com/subscribe
Learn the strategies we discussed in this episode - head to https://pages.thequalifiedanswer.com/profile for our full training program.
Want cool merch? [NULL]Shirt: https://shop.nullshirt.com/
G'day. Welcome back to the Qualified Answer Podcast. So I'm Alicia here. Thanks so much for joining me. And this episode, it's a solo episode, we're going to cover three interesting facts or points or perspectives that might help you in your day today. The first one is one of my favorite topics to rail against, which is around the topic of using averages in your measurements. I'm not saying you shouldn't use averages, but I find that averages are average. Averages tend to hide all the interesting detail in any data you're looking at, particularly performance data. So really what you'd want to be looking at is things like the 99th percentile or the 99th point ninth percentile or the 90th percentile or the 80th percentile, something at the extreme range. The reason for that is you want to see what the worst experience is like, not what the average experience is like. You know, if I have 10 experiences of which one is bad, I'm gonna remember the one bad one. If I have millions of users are rotating through a system, the likelihood is they're gonna hit the tail end bad experience at some point. This also has some interesting math when you add many components into the system, it will also start to magnify the impact. But really, what it's about is focusing on the things to pay attention to rather than getting lost in the noise. So focusing on the 99th percentile solves for a few problems. Firstly, it means you're seeing an extreme effect, but not the most extreme effect. Because you know, stuff happens. There's no point looking at the most extreme effects. So you want to kind of cut them out. That's that 1% you're cutting out. But when you start to optimize and look at what's happening at the extreme end of the continuum, the changes you make will typically also have an outsized effect on the rest of the continuum. You know, rising tide raises all ships. By improving performance at the worst-case scenario and maintaining or improving performance at the other end of the scenario, you're improving the system as a whole. By using this approach, it also helps you dive deep quickly when you need to. So rather than looking at a sea of averages that all average and look okay, this lets you dive in very quickly. Now, I mentioned the 99th percentile being useful. Again, it's a classic, it depends. Sometimes it can be 80%. That might be the space you want to look at. It depends on the use case and what's going on. If you're talking very technical, uh technology-focused code performance, very hard mathematical metrics, and yet P90 is and P99 is really good. If you're talking about more squidgy human issues and uh things that have human factors involved and stuff that's not quite as precise as it could be, well, often the 80th percentile, 90th percentile is where the gold is. So the first tip today is if you're only using averages, you're probably missing out on a lot of detail and a lot of information as to where to look to fix things and make things better. So it's a make your life easier tip is the first one. The second one I want to address is the current phase of what I would call AI washing going on in the industry. And we see this all the time. I lived through the cloud washing days. You know, I was working at AWS and we were the cloud at the time. And it was amazing all these companies around us who suddenly had cloud in their name or talked about cloud, and they hadn't changed their product, they hadn't changed the way they build, they hadn't changed the way they provide a service, nothing changed, but the market expected to see cloud. And so the reason for this is pretty straightforward. Firstly, if you're leading in and actually doing AI, well, of course, you're going to have AI in your name. If you are a competitor organization and you're being left behind or caught out, you're going to want to put AI in your name because you're trying to draft off that interest and off that market need. Often the market, if you're a publicly listed company, will expect you to have an AI play, even if you didn't see it coming. Thirdly, I'm seeing a lot of strategy in board papers that are sort of, I would call AI washing what they're trying to do as well by just using the word AI a lot, banking on the fact that most people don't understand what it actually means, and trying to sort of skate through and get some funding because that's where the funding is going at the moment. The reality is, is firstly, is AI in many ways is a feature, not a product. So what I mean by that is AI gets built into things to make the thing better. I said very early on in this AI revolution that AI will disrupt every knowledge-based value chain. And I've been proven to be right over and over again on that topic. Essentially, if you're seeing a business process or value chain being manifestly changed by artificial intelligence, maybe it's made far quicker. Maybe it's done more simply, maybe integration is faster, maybe the customer experience is completely transformed. All those types of changes are real AI. Something with a chatbot added to the side of the web page, not AI, in my view. Bit of AI on the side there, but it's not doing anything to the to make the change. And then what always happened with cloud washing is people say, well, I've moved to the cloud, or what I thought was the cloud, and I'm getting no benefits. And I had to deliver the uncomfortable news that, well, you haven't moved to the cloud. You've moved to something that's calling itself the cloud, but it's not the cloud because it doesn't represent the things that changed that were driven by the cloud type technology. Same thing with AI. If you talk to folks and say, well, we added AI and nothing changed, it's like, well, talk to me about the business processes that were changed and the way of working that was changed and the customer interactions were changed. Oh, no, no, we just added a chat bot. Well, that's not AI. That's AI washing. And nothing good comes of that except a lot of wasted time and a lot of wasted money. And the third one I want to talk about in this episode is around tooling and the tools that developers use. Now, everyone is different, everyone has preferences with tooling. But I think it's really interesting that I'm seeing a shift at the moment. I'm experiencing it myself where the IDE, the development environment that the developer lives in, was really, really, really important until it wasn't important. And what do I mean by that? Well, I would spend hours tweaking my fonts and my shortcuts and the way files opened and the layout, et cetera. And people far smarter than me had way better extensions and had build extensions, and the IDE was the thing that did all the things as a developer. And it was great. I remember growing up in the time where you literally used the text editor to edit your code. So IDs were absolutely awesome. A 10X step change, you know, built-in debugging and project management and control and all the all the good stuff. And so we would live in that space, in that that was our environment most of the time. But if you've adopted a very AI-heavy, slash agentic approach to software development, you'll find yourself reading a lot less code, a lot less frequently, and spending a lot less time in your IDE. In fact, I'm finding myself spending no time in my IDE. Spend a lot of time still in my text editor. So Zed is my current preferred text editor, it changes all the time. I was a big sublime text person for a long time. Uh, but now I've moved to Zed. I like its speed and its flexibility and all the good stuff it brings. But really, it's very occasionally I'm jumping in and actually reading code because most of the time I'm interacting with my AI harness to say, hey, let's reason about this code, let's talk about this code, let's uh inspect the code together. And it's basically become my IDE plus plus plus. And this is not to say if you're using an IDE, you're doing it wrong. You're not. You get to choose, you get to do it exactly how you want to do it. But I think it's an interesting trend when we see tooling that was fundamental to folks disappear or get drastically reduced in its use. And what that typically says is again, the value chain, the way of working is fundamentally changing. Let me ask you this question. If you were setting up a brand new business, software business yourself today, would the space of developing a new IDE be the natural place for you to go? Probably not. Probably not. And we're seeing this interesting blending. So things like Wes Term, for example, you know, it's a terminal with a gentic AI and a bunch of stuff. So people are trying to figure out how things fit together. But I would suggest that the days of the IDE being the king of the heap from a developer standpoint are over. Now, saying are over sounds big and scary. It doesn't mean, well, tomorrow no one's using IDEs. They'll be used for many, many, many years to come. Nothing, nothing ever goes away. But I think what you'll find is that people will care about their IDE a lot less. And the agentic harnesses and software that is even yet to be built will become the new way of working. One of the interesting related trends is the use of voice to be providing prompts and to increase speed because you can speak, I think, three times faster than typing. I've tried it twice and it hasn't stuck for me. I'm on my third try now because you know the technology has moved and shifted, and I'm seeing, well, does this work for me? So interested if you're finding it useful as well or not. Um, and I think it's contextual as well when it makes sense or when it doesn't. But again, for a long time, folks like Simon Wardley have talked about conversational programming and the ability to just talk about the problem. And I can see the benefits. It is a different way of thinking, though. It's quite interesting to be reasoning in your head as you're speaking. But I think there's a there there, something there to be had, but I haven't quite figured out the ergonomics right. And I think this really becomes a question then of developer ergonomics. What makes sense to you as a developer? What do you like to use? What are your preferences? You get to choose. Everyone's different, everyone gets to make a decision. But I think the consensus is moving away from the IDE as the center of gravity. And for a developer, that's a seismic shift. That's a real seismic shift. It's kind of like, I don't know, a blacksmith not using the anvil anymore, for argument's sake. You know, it's a pretty big shift in the way we work. Where it's going to land, I don't know. It's an observation, though. When I start to see key components of any value stream and value chain being fundamentally altered, that's a warning sign and a sort of alert signal to pay attention. Something's happening here. Uh, we just need to figure out what it actually is and see how it plays out. So three different but somewhat related topics today that hopefully have sparked your thinking. And hopefully you've enjoyed this episode and got a little bit out of it. Of course, comments uh by the link in the show notes. And until next time, keep on building.