Code and Conscience: The Logic of Trust

Transcript

Code and Conscience: The Logic of Trust

It is absolutely culture, not code. Organizational inertia is an incredibly powerful force. And to actually get to real outcomes, real adoption, real scale of any of these tools, but automated reasoning in particular, you have to actually change the way people work.

My hope is for leaders to increasingly rely on automated systems that they can trust for a vast number of decisions so that they can spend more time on bigger decisions where human judgment is a unique value added.

What we should be teaching the next generation is this form of discipline judgment. You should know when to lean on the data and when to trust your read of the room. And also when they're both in tension and why. So here's the thing about AI, right? It's trained on history, it's trained on data, but good leadership is often tested when...the history breaks and those moments of newness arrive.

It's 2026. You're the CEO of a global bank or the head of surgery at a major hospital. An AI system looks at the complex set of volatile market data or a patient's decade long medical history and gives you a directive. It doesn't just give you a percentage of probability, it tells you exactly what to do.

But here's the catch. If that decision fails, the machine told me so, won't hold up in court, in the boardroom, or at a patient's bedside. For a few years now, we've be we've played a game of black box roulette. I'm gonna do that again. For a few years now, we've played a game of black box roulette, using AI that predicts the future based on the past. But in a world of sudden market shifts and unprecedented global change, the past is no longer a reliable map.

We are entering the era of automated reasoning. This is the shift from AI that guesses to AI that proves. It's a market for explainable systems that is set to hit nearly $10 billion this year, doubling by 2032. Because today, leaders don't just need an answer, they need the logic behind it. They need a glass box they can trust.

Welcome to the Next Five Podcast. I'm Tom Parker, and this is the first of a three-part series we're calling Code and Conscience. Over the next three episodes, we're exploring the $4.4 trillion AI opportunity by looking at the rules of data and the weight of human judgment. Today in episode one, we're tackling the logic of trust. Joining me are three leaders at the forefront of this mathematical and cultural shift.

First, we have Scott Wiltamuth, Director of Software Development for Agentic AI and Automated Reasoning at AWS. Scott, welcome.

Thanks it's great to be here, Tom.

Next, we have Mary Martin, Managing Director and Senior Partner at BCG, who leads their North America tech and digital advantage practice. Mary, thanks for joining us.

Thanks for having me, Tom.

And finally, Varun Chitkara, Senior Vice President of Global Product and Technology at ADP. Varun, it's a pleasure.

Thank you,Tom. Excited for the conversation ahead.

Well, Scott, let's start with you. We've all become accustomed to generative AI that feels like magic, but it can hallucinate. You're leading the charge in automated reasoning and neurosymbolic AI at AWS. For a lay person, what is neurosymbolic AI and what is the fundamental difference between an AI that forecasts an outcome and one that provides a mathematical guarantee? And and how do you

Build and test these systems.

Well, I'll start with automated reasoning. So automated reasoning is a field of computer science that attempts to give assurances about what a program will or won't do, or what is true or what is false. And if you think back to eighth grade geometry or something, you probably remember trying to, you know, prove what the circumference of a circle is or what the area of a circle is, where it's a step by step.

proof that starts with a set of facts and then uses logical reasoning to reach a conclusion. And that's essentially what automated reasoning is, just at a much larger scale and with a much higher degree of automation. And it's a fantastic counterpart to machine learning. With machine learning and gen AI, there's always an error rate. And you know for Gen AI it shows up as hallucinations.

And for automated reasoning, there's never an error rate. You may have false assumptions, but like you're always going step by step and proving things about the systems. And so we see the combination of those, the neuro and the symbolics, of neuro symbolic AI, as a very powerful combination of those two technologies that are kind of at opposite ends of a spectrum.

Brilliant, thank you. Mary, this year we're seeing a near twenty percent growth rate in AI that moves beyond prediction to actively recommending actions. Are leaders asking for this transparency because they want better results, or is this a defensive move to satisfy regulators and auditors?

Yeah. Tom, the single most common question and topic of conversation that I have with enterprise leaders, executives, really revolves around how do I get some confidence that the investments I'm making in AI are going to deliver value. That's their starting point. And they are feeling that the need to invest, they are making greater commitments if you look at at some

Recent findings, the majority of CEOs would say, I am on the hook. I am the chief AI officer. This is a huge enterprise priority. But they feel the gap in confidence that it's going to return value. And when you then map that to like, well, what is the range of technology available? They are absolutely looking at how we follow the path of advancement in AI solutions to continue to drive.

To greater value creation for the enterprise. But those enterprises that are getting the most, that are getting the highest returns and the best confidence in those returns are not just applying technology for the sake of the best, latest, greatest instantiation of the technology. They're starting by looking at where are the actual opportunities across the value chain? Let me be prioritized, let me be focused and selective. And then they are designing solutions that actually meet the needs.

Of those value pools. And those solutions may be more quote-unquote predictive, they may be more deterministic, they may be more advanced reasoning depending on the need, but it is that fit-to-purpose kind of construct that is actually allowing leaders to gain confidence that we are making the right investments for the right return profile. The transparency piece and that kind of defensive layer then becomes a part of the story of scale and value assurance. Because as I

and making these investments, I need to know that I'm not taking on undue risk, that I can see how the reasoning is happening, particularly in highly sensitive parts of the business. And so that I see it as a necessary requirement to continue to scale and to continue to have some confidence in the value creation.

Yeah, absolutely. Varun, Mary said about this scale and value. You've written extensively about avoiding sophistication bias, the trap of using high tech tools where they aren't needed. How do you distinguish between a problem that requires a mathematical certainty and one where human judgment is still the better engine, if you will?

Thank you, Tom. So this framing of sort of certainty versus judgment in and of itself may not be a true choice. Most of the interesting problems sort of sit somewhere in between in that space between them. So when you begin to think about problems, I think of it in three buckets. First, you find these repeatable, well-defined patterns. know, examples would be payroll calculations, fraud detection, inventory reordering. These are prime candidates for automation.

Mathematical certainty is exactly what you need there. Then you go to the second category, which is sort of the middle ground, business forecasting, scenario modeling, risk modeling. AI can help you really sharpen the thinking, do some calculations really fast, but that final call really requires that human context and judgment. And then you have the third category where you have decisions that I believe should remain deeply human, know, crisis, leaderships, layoffs, ethics, trust.

These involve organizational dynamics, human emotion, consequences that extend well beyond what the data may tell you. So the underlying principle in all this, and that's what I wrote about in the sophistication bias, is this idea of fit for purpose. As organizations, we tend to reach for the most advanced solution that we can find, whether or not the problem actually warrants it. If you think about it, some of these things may actually do need advanced AI. Others may be simply better workflows, clearer articulation.

cleaner data, et cetera. So all in all, if I look at AI, it's exceptional in sort of reducing that ambiguity that may exist in the data or the insights. But when it comes to values, humans are still better at navigating those. So when you have to apply this, I would say as you move away from efficiency and financial considerations more towards consequence and implications, that's where human judgment matters more.

I want to move on to the stakes that are involved in the integration of AI with human capital. In healthcare, Scott, we're moving towards longitudinal insight engines. When you work on these systems, how do you ensure that AI isn't just finding a correlation? Like people with umbrellas are more likely to have wet hair, but is actually understanding cause and effect.

well I I think if you're talking about that there's potential for error rate, that's kind of really in the domain of the machine learning. And I think in general it's not possible today to use automated reasoning to just you know have a general purpose like hey for all domains I can say what's true and what's not true. Like it's not possible to do that. But if you scope down to a particular domain,

That can be reduced to a set of rules. So Varun already mentioned, you know, payroll, payroll processing or taxes, or or something of that nature. You can define that domain in a way where you can, similar to my geometry example, proceed step by step from a set of facts and then applying rules and then reaching conclusions about them. Now, in the healthcare domain, that's unusually difficult because the human body is.

Extremely complex. So that's not a domain that I would tend to go after with the lever of automated reason. So I'll agree with Varun the right tool for the right job. So what domains are good ones to go after? domains that have a lot of rules, so HR policies, tax law, regulatory compliance, et cetera, are good examples where you can scope the domain.

Have a domain expert define the domain, which is like literally a piece of software that reduces the domain to a set of rules. And then you can test that domain. And this is exactly the kind of thing that we do with automated reasoning checks for bedrock guardrails, which is a feature in Amazon Web Services, where you literally start with a blank canvas and then can ingest a rules or a large number of documents, et cetera, that define the domain. And then you can have an expert.

Test the domain, look at corner examples, et cetera, and then say, yes, I believe that this is accurate. And then use that in practice and see how it actually works. And see that you know, build trust in that system. as a general thing, like you won't see one of these for Chat GPT or similar use cases where it's a very broad set of domains. Gonna be more targeted where you can scope it and reason about that particular.

Yeah, absolutely. I mean, you said there build trust in the system. That's essentially what this whole episode is about. And Mary, I want to look back at healthcare a bit here. What are some of the high risks that trusting an automated reasoning AI can cause in the healthcare space? And how can we protect against this? And are there other industries that are high stakes where the risk is so big that we can't even consider the outcome of failure?

Yeah. Let me maybe start with the second part of that question, Tom. And I I do think anytime we are looking at industries or sectors, or frankly, workflows that have a deep and direct impact on human lives and livelihoods. So healthcare comes to mind, financial institutions, wealth management, critical infrastructure, things that are going to touch kind of the day to day life of humans. It is natural for us to think about risks, and there's a lot.

That can be done to mitigate those risks. I think we would be wise, though, to also just pause for a moment on healthcare and remind ourselves that this also represents a massive opportunity to raise the floor on the level of care. I think a lot about this current moment in time with AI, and I think about the potential living at the point at which we can break compromises that we have historically had to accept.

in the standard operations of any sector. So for healthcare, when you want to deliver one-to-one patient care in a way that feels intensely personal, meets them in their moment of need, but you want to do that in a scalable way that is not so resource intensive. That's been a compromise in healthcare. And I'm sure most of us have experienced that at one point or another.

The potential of the tools and the technology as it exists today is to help us alleviate some of those compromises and better meet the needs of all the stakeholders. So I do think it's worth reminding ourselves that even as we tackle risks, we are tackling them in service of creating opportunity. On the risk side specifically, in spaces like healthcare, it's not just that the you know the machine or the AI might inform one bad decision. You think about one.

Wrong diagnosis that has, of course, a huge impact for that patient. The real risk is that there may be a scalable challenge, right? That the at scale, the AI is informing a series of bad decisions, and there wasn't a system built into place for human intervention to help mitigate that risk. That's the real, the real risk that we're tackling. What I'm seeing is a few layers of mitigation to that. And I think it starts kind of with what Varun was saying earlier of have we focused on the right places where the technology is sufficiently mature and the risk can be mitigated that we should invest to break those compromises while other pieces perhaps are continuing to mature? So that prioritization and focus is the first layer of mitigation. The second layer of mitigation that I see often, especially in healthcare, is are we designing the system in a way that we're using not just general purpose models, but fit-to-purpose models? Are we

Tuning the engineering that underpins it so that the agent itself has enough awareness to say that is not a problem I am ready to tackle or that I can tackle, right? That we have actually engineered in a way to design for risk mitigation. But the most important layer of that mitigation is have we designed the human agent, human AI operating model, not as an afterthought, not as something we set in place later when we realized there was a risk, but have we designed it from the beginning.

with the right points of human intersection, the right decisions that are allocated to human judgment, and the right way that the AI assistance is getting into human hands over the course of the workflow, right, to inform better outcomes and to help to kind of structurally mitigate the risks.

Yeah, absolutely. Verun, Merry said there to the giving decisions to human judgment where we choose that. I want to push a little bit more on the ethics of this because trust is an emotional state, but verification is a technical one. And I'm imagining a scenario where a colleague makes a mistake and when looking at the consequences, we factor in a variety of human and emotional biases. You know, I quite like Bob.

you know, he's not been himself lately. There are family issues. I think we should let him off this one. These are the types of conversations we've all had. So how can we give leeway to the human emotional state that caused a negative outcome, but not to a machine? And why do we justify the mistake differently when it's easier to code out the problem in AI than perhaps to rebuild trust in a person? And then there are a few questions here, but at the top of the show.

I said that the machine told me so isn't a reasonable defense, but is Bob told me so still a valid excuse? or do we have to rewrite explainability into a human right as much as it is into a machine's code?

Yeah, no, as you said, this is a multi-part question. So let me try to sort of take this on. I think at the core, it boils down to how we define trust. When you think of humans, trust in humans is relational. And when you think of machines, it's more transactional. And that distinction almost explains everything that you have to think about, and how we respond differently to Bob, versus that of the machine. When Bob gets it wrong, we're not just looking at the outcome, we're talking about context. Was this isolated? Has he done it before?

Was he under pressure? Did he have the right information? And we put all of that in context. So humans sort of are existing inside these relationships, history, social context in ways that machines don't. But here is the irony of this all. Humans are often less explainable than machines. Leaders make decisions all the time, and we try to rationalize them afterwards. But that intuition, the word that you used, is sort of pattern recognition that built

that's built through decades of lived experience. It is also a form of reasoning. It's just not one that we can clearly articulate. So when I think about how this plays out, maybe it's less or maybe not just about humans demanding that the AI explain itself. Maybe it's also that humans now need to explain themselves with the same rigor. So if you put this all together, maybe that's very healthy, right? And if the collective goal is to get to better outcomes, reduce harm,

while still sort of preserving that empathy or accountability. Maybe we shouldn't look at it as humans versus machines, but more about how do we build the systems where both become more transparent and more trustworthy.

Yeah, absolutely. I'd love to open this question up to to Scott and Mary. Scott, have you got anything that you would like to to come in here with?

I do, yes. I think you know, for agentic systems the error rate comes in the forms of bad bad behavior. And so this is something we're working very hard for kind of getting the building blocks in place where you can prevent agentic systems from specific bad behaviors and have those, you know, verified or proven with automated reasoning that you can have like a hard boundary. And I'll give a an example which I think is helpful.

We're thinking about how dealing with humans and how dealing with software-based agents are really different. So, you know, people don't really use travel agents much these days. Like that's kind of become a highly automated area. But I'll use it as an analogy. So if you're planning a trip and you give your credit card to a human travel agent, there's kind of an implicit contract in in that, which is, you know, hey, you can use this credit card to book trips for me.

You know, that I've specifically told you about, like, hey, I want to plan this trip to Paris in August. And you can't, you know, buy yourself a painting with that because that's not what I gave you the credit card for. and for human relationships, they're kind of also implicit guardrails that exist for that. So that travel agent wants to get a referral from you, they want to get

Repeat business from you, they don't want to go to jail for credit card fraud. They don't want to be fired by their agency. So you have a natural kind of implicit set of guardrails on that transaction. But if you're dealing with a software-based agent, the software-based agent doesn't fear any of those things. At least they don't, they don't yet. And so it's helpful to have a set of guardrails. And expressing those guardrails in software is something that is very amenable to the application of automated reason, where you could, you can, you can put

You know, restrictions on it, like what is the time frame that can be used, how much is the limit? Is it okay to book things that are refundable or non-refundable? You could have a variety of rules that kind of bound that and have essentially a different set of guardrails for the software-based agent than you would have for the human-based agent. But to get the same net result, which is that you would like to have certainty that it's okay to hand your credit card over. And indeed, with the software-based agent, I think.

We'll have stronger guarantees for that than we do with the human-based agents. Because humans, as Rarun pointed out, humans are more like machine learning. They always have an error rate. And there's no need for that in software-based agents, that we can put guardrails on that to prevent bad behaviors. And then, you know, as a more advanced topic, you can also talk about well, how would one get use automated reasoning to get good behaviors out of your agent to kind of specify what the result is of that? But I think as a starter,

Using automated reasoning to prevent bad behaviors with guardrails is a very powerful model.

Mary, what are your thoughts on Bob versus the agent?

Yeah. They may be more similar than different. And I think there's some things we can learn about how to manage the agent that we see in organizational dynamics. I mean what we're talking about here is really accountability diffusion. And it is more diffuse when we're talking about AI-based systems, because it's a little harder to see exactly where did the

know the decisioning beginner end or who should be held accountable. But it happens in organizations as well, right? Bob was doing something his manager told him to do, informed by someone in another function. Ultimately in an organization, you have a a chain relationship and there is someone who ultimately is going to be held accountable for what happened in their function. And when we think about putting in place the right governance and responsible AI management around AI systems,

One of those key principles that we think about is there is ultimately a human who is responsible for the agents that are operating. There is a human that is responsible for what data was allowed to be exposed to those agents. And having that clarity, and it kind of comes to what we were saying earlier, having it built in from the beginning of how are we going to govern decisions about agents, decisions about data, decisions about access rights.

Down to like who is the human that is named and is accountable is a key part of setting up a framework that allows these types of systems to scale responsibly and securely within enterprises.

Absolutely. I want to start looking five years out. Scott, you oversee the AWS Machine Learning University. How are we training the next generation to build these glass box systems? And is the future of coding actually more about philosophy and logic than it is about syntax?

Well, first, just about Machine Learning University. So most of Machine Learning University is focused on helping Amazon builders kind of accelerate the adoption of machine learning and other techniques within Amazon. We do have a small portion that's kind of focused on external, but it's mostly about our internal builder population of kind of bending the curve of adoption. Like we anticipated when we founded Machine Learning University, maybe.

eight years ago or something, 10 years ago maybe, that machine learning would become infused in all software. And we wanted to prepare our builder community within Amazon. So software developers, applied scientists, technical program managers, et cetera, to be able to apply this technology and apply it in a responsible way. And so we thought that if we could accelerate that adoption, it would be a strategic advantage for Amazon to move fast in this area.

And that's what we focus on through education. We really think about it not as you know, is is everything trending toward math and logic? It's more about educating our builders about the wide variety of tools that are available to them and helping them think through what it's like to build systems with these advanced capabilities. And that includes a healthy dose of responsible AI, which we cover in every course that we have.

Or what are the impacts on humans of these systems? And how does one build systems and kind of can conceive of and build systems in a responsible way that are gonna generate business value for Amazon and have positive impact on all the people who interact with those systems? and that that's kind of been our focus. So it's really using education to help people understand how to use a wide variety of tools, not just machine learning, but machine learning and

Robotics and automated reasoning and and and to build these advanced systems in a way that delivers value and is responsible.

A follow up for both Mary and and Varun here. In your careers, you've likely made decisions that felt right, even if the data was messy. If an explainable AI provides a logically sound recommendation that contradicts a leader's intuition, which one should we be teaching the next generation of managers to follow? Varun, let's start with you.

Sure, this is a tough one because most important and good decisions sort of have both, right? They're informed by data and by intuition. So in short, what we should be teaching the next generation is this form of discipline judgment. You should know when to lean on the data and when to trust your read of the room. And also when they're both in tension and why. So here's the thing about AI, right? It's trained on history, it's trained on data, but good leadership is often tested when the history breaks and those moments of newness arrive. An example recently is COVID. When COVID hit, every historical model became a very weak predictor of what to do almost overnight. At that point, what mattered was adaptability, judgment, values. And then fairly quickly, we went back to data again. You know, when we had to do the six feet social distancing or the efficacy of different types of masks and the vaccine distribution system, all that relied on

mathematical certainty, data projections. So the combination is sort of what brought it together. Now the leader's intuition, it's not just instinct and off the gut, right? It is seasoned that the pattern recognition that's built over years and years of actually having gone through that. It deserves respect, but it is also something that's very subjective bias and we should be honest about it. So, you know, to your question about what do we teach? We shouldn't blindly trust AI. We should also not blindly trust the gut. We should sort of teach this dual discipline of holding both while being intellectually honest about what each can and cannot do. And when we do rule one over the other, why we do it and with what context.

Varun trusts no one and nothing. That's essentially what you're saying. No, I jest. But Mary, let's open it up to you.

Yeah, I would agree with everything Bruin just said. I was actually Tom recently looking at some data from a study that Wharton published. And it was comparing the accuracy of answers and the confidence that humans had in those answers when they reached the conclusion on their own, when they reach the conclusion supported by good AI, and when they reach the conclusion supported by bad AI. And as you would expect, right, the accuracy of the answer increases supported by good AI.

Decreases from the human level or the human benchmark when you have bad AI in the mix. But really, really interesting. The human's confidence in that answer shoots up and is consistent, whether it was supported by good AI or bad AI. Humans are very confident in answers where there is a logical, rational, right? Feels like it could be true explanation presented to them. And I think it is a really good.

Proof point for why we need exactly the training that Brune is describing for our leaders: of how do you bring judgment and intuition into the mix? And it's not only judgment and intuition of having seen more cycles, of having weathered some of those moments that required adaptability. I also see that the best leaders are able to look at things that seem like outliers, are able to hold it with a curiosity.

That allows them to interrogate is this truly an outlier we should write off, or is this an outlier that is indicating there is a different opportunity that we should be examining? And the models, just as Varun described in the way that they're designed, the way that AI AI is meant to work, tend to kind of write those things off because they lie outside of the norm and the pattern. The really good leaders are looking not just at patterns, not just at adaptability over cycles, but also at what else.

might be opportunities that we haven't thought of yet or that we are just beginning to see that could be new creation, value creation opportunities.

Thank you, Mary. Varun, back to you. If we use automated reasoning to justify decisions based on historical data, how do we stop it from just finding sophisticated ways to logic our way into keeping old biases alive?

This risk I think is not just that AI gets it wrong randomly. The risk is that it gets it wrong systematically and with complete confidence. I've referred to this in past as plausible nonsense. It's very believable, but it's sort of really not based on anything. So the core issue here is context. Intellectually, honestly, if you look at it, it requires us to keep on asking the question, is the world the same world this model was trained for?

You referred to it in your opening too, but we really cannot solve tomorrow's problems with yesterday's tools. And this is sort of the same challenge we face with people. You know, when an employee stops learning, works in silos, stops upgrading their skills, they sort of become a liability. We don't let their work go unchallenged. There is supervision, is, you know, checks on top. So we should not treat an AI model any differently. If I talk about it on the technical side, or you've heard of knowledge graphs and context graphs,

they can bring in real world relationships in current context. So the model doesn't just stay a pattern matching on sort of stale history. But here is another idea worth considering, right? If you begin to think of agentic AI as a member of the team, whether it's different agents, different personas, different lessons, on the same problem, that diversity of perspective is how we can catch what a single model or in fact a single human may miss.

So if we keep testing the data, the assumptions, the human systems underneath, then we can get rid of it because bias isn't really sitting in those algorithms. It lives in that context that's sort of outside of it, right? So to sum it up, if I were looking for an antidote to this bias, I would say it's sort of the same intellectual honesty context and sort of the diversity of opinion, which we would talk about even without AI.

Thank you so much. Mary, how do we, or how do you convince a board of directors that a more transparent, complex system is worth the investment over a cheaper, predictive one? Would would they not argue that they should invest more into people and build trust and logic into humans first?

Yeah. I spend a lot of time with executive teams and with their boards and investors on the topic of how do we get return, how do we optimize return out of this investment? And I might offer Tom, I don't think it's an either or or a first and second. It is both and and and in parallel. And what do I mean? Like any other large investment, investments in AI.

Solutions that are gonna change the business model are they're capital programs. They're capital intensive, they require a kind of transformation, dedicated focus. And most organizations that are moving at pace are evaluating them through the same lens and the same rigor that they would put on any other capital investment. And in fact, the return on that capital for most of them is a higher rate of return than other places where they might deploy their capital. So they're very good investments.

But like any capital investment, what you're trying to do is get to the highest risk-weighted return profile and the greatest confidence that you're going to actually see the return. The tooling is, as we've described throughout, a fit-to-purpose. What is the right solution to drive the return profile? The transparency is a mechanism to de-risk the returns, particularly when we're operating in some of those highest risk areas where we need to be.

Very confident that the solution is going to be explainable and transparent to people. So it is a mechanism for actually creating that risk reward profile that a board is looking for. On the people side, we see over and over, no matter the sector, no matter where in the organization or value chain they're deploying AI, that those that are seeing the highest returns are the organizations.

That have systematically tackled how are we bringing people along. Again, not as an afterthought. Let's go train everyone after we have everything rolled out. But actually, how have we from day one designed the new operating model, the new talent and skills profiles, the upskilling, the new teaming models, the new organizational structure? All of that needs to be put in place as the technology is built because that's what locks it into the organizational DNA and allows for you to really see sustained results. So they have to happen in parallel to actually deliver what the board is ultimately looking for, which is a strong return.

Brilliant. Thank you, Mary. Well, before I let you go, I want to do a quick fire round for all three of you. firstly, what's the one roadblock, technical or cultural, that could stall our ability to build trust and logic into an automated reasoning future? Scott.

Well, it's interesting. Until now automated reasoning has had what I think is like a 0.01% field. Like it's only like the rare software project, like literally 0.01% or perhaps less, that has ever had anything proven about it. And we're on a mission right now to get four orders of magnitude better than that, to have a hundred percent of software develops be better in some way because of automated reasoning.

And in many cases, we can do that without having deep involvement from the software developer. We can just make the tools better and we can kind of have this kind of magical combination of the machine learning with the automated reasoning to make things better. But if you want to get true proof, if you want to prove aspects about your software systems, software developers are going to need to learn to specify what are the good outcomes that they want, what is the good thing that they want, not just the bad behavior that they want.

And I think the biggest barrier I see to getting to a hundred percent or kind of toward a hundred percent for proving things about our software systems and a about the positive attributes of them is developers' ability to specify exactly what they want as a result. That's the biggest barrier.

Mary

You have to change the job they were doing into something you want them to do instead. You have to change the points of handoff. You have to change their knowledge base. And that is counter to the underlying inertia of the organization, which moves on a quarterly or monthly or weekly cadence. And so actually attacking that change management with focus and intention, even more focus, I would argue, than you're putting against building the right code base, is a critical, a critical unblock.

Varun, you were nodding along there.

I couldn't agree more. And I was also smiling with the question because this is probably the toughest one you've asked me. I've written about seven blockers that reinforce each other and you asked about one. So I'm going to try to sort of take a different tack at it. I think there's a single human trait underneath all seven and it is fear, which shows up in two ways. It's either the fear of missing out, which drives organizations into sort of this frenzy of hyperactivity.

without the right foundation, controls or discipline. As a result, accumulate failures and fail business cases, which then sort of take it off the rail. The other side is exactly the opposite. It's a fear of consequence. So when people think too hard about how to do something, they end up at this sort of analysis paralysis state where we don't take any action at all, right? Both of those sort of lead to what Mary was pointing on, this sort of idea of organizational readiness. And I think that becomes the real blocker.

And in many ways, it sort of shows up because AI can generate answers faster than humans can probably absorb, comprehend, validate, or implement. And that asymmetry becomes a limiting factor. So in short, the limiting factor is not the technology. It is us. And maybe the question we shouldn't be asking, is the AI ready for this? We should be maybe asking, are we ready for AI as humans?

Absolutely. I mean, Varun, you just touched on it there that AI can create answers and log logic quicker and faster and better than perhaps we can comprehend. So in five years' time, will the average person feel more secure because AI can explain itself, or more alienated because the logic is too complex to follow? Scott, back to you to start off this one.

I think it's gonna be a mix. I think in domains where that are regulated or more easily reduced to rules, I think people will have more confidence. So for agentic scenarios where it's reasonable to specify the guardrails, I think people will have higher confidence. I think in general for kind of like very broad general purpose use.

Like is popular now with you know Claud and ChatGPT and others. I have real fear that it'll be worse and worse, that we'll have a real problem with confidence and also some real problems similar to problems we've had with social media kind of, you know, negative impact of those technologies. And so I think a real challenge for the industry is to have the kind of

controls in various forms outrace the you know the risks of having error rates and having the error rates be in hallucinations and bad behaviors and things. So I think it's a little bit of a race. But I think in specific domains I think there will be higher confidence. But in very general purpose use cases, I think I'm I'm it makes me nervous and I would be more pessimistic.

Mary.

Maybe if I take it into an organizational context, Tom, I'm quite optimistic that transparency is going to enable the average employee to feel more secure. And I I think about work that I've seen where we are upskilling humans to have this kind of very human AI assisted model. And we hear things like, This is the most fun I've ever had in my job, right? Because they can.

Go farther, they can tackle challenges they weren't. They see how they're breaking some of those compromises that we talked about earlier. And that is a real sweet spot, right? As the transparency and that human agent, human AI relationship builds, I think that can really unlock actually a lot of human joy and creativity. I think if transparency and explainability simply becomes another checkbox on a compliance document and cumbersome.

To humans being able to tackle their goals and objectives, it does run the risk of creating alienation, but I don't think it has to.

Varun.

Yeah, no, I agree with what both Scott and Mary have said, right? And the answer is honestly both. And this isn't sort of meant to be a cop-out, right? I do believe it is the most accurate answer. And we've seen this before, right? If you think about, you know, when calculators arrived, some people felt empowered, some people felt threatened. Excel came and replaced spreadsheets. And we were talking about some people who got in and really mastered it and others sort of who used it to sort of accelerate their careers. There were people who didn't touch it at all and they got left behind.

It was the same technology, but different outcomes. We also see this at an organizational level. You think of Nokia and Blackberry. It wasn't because their technology was bad. They just could not bring themselves to embrace what was changing. And the organizations that leaned in, even if they did it partially, they found new ground and they made a lot of success. AI is much bigger than any of these. But I think the underlying dynamic is very, very identical.

There'll be two types of people, right? People who engage, they will accumulate small wins, they'll build familiarity over time, and they will feel more and more secure and confident with doing this. The same extends at an organization level. On the other hand, people will resist it, they'll be on the sidelines, and they will start getting left more and more behind. So if I think about it, you know, elevating it to sort of organizational or even societal level, it all comes down to how much we invest in bringing people along.

and getting them to try and feel more confident. Because left individually, you'll end up in both, and people will have very strong opinions on either side.

Okay, well l the last question here before we leave. What is your one hope for how automated reasoning changes the relationship between a leader and their data in the next five years? Scott, back to you.

I think my hope is for leaders to increasingly rely on automated systems that they can trust for a vast number of decisions so that they can spend more time on bigger decisions where human judgment is a unique value added.

Mary.

Very similar. I work with a lot of leaders who are data rich and insights poor. And they spend a lot of time wrestling through messy data to get to answers that are basic answers of what happened and maybe answers of why did it happen. And there is very limited time to spend on so what do we do? And so my hope is that as you can automate more of that, those layers of what and why, you can focus.

The human judgment, the human intuition on the so what and how do we go from here, which is really where I think ingenuity shines.

Varun.

Yeah, so my hope is that AI doesn't change what great leadership looks like. I think it should reemphasize some of the core values. We spoke about some of those today, right? The intellectual honesty, the discernment and knowing what to use when. To be authentic to know when you can use something versus something else. I think that's going to be key. So all in all, you know, this new definition is sort of the old one, but on steroids. And overall, I hope this gives us the space to be more human, not less.

Well, it's sadly time to end this episode, but what a fascinating one it's been. My thanks again to my guest, Scott Wiltamuth.

Thank you very much. Thanks for having me, Tom.

Mary Martin.

Thanks, Tom. It was a delight to be here.

And Varun Chitkara.

Thank you Tom for having me. This was exciting.

Well, as we close this first chapter of Code and Conscience, I'm struck by a recurring theme from our guests. The idea that we're moving out of the age of intuition and into the age of evidence, or more importantly, a balanced mix of the two.

It's easy to look at the four point four trillion dollar AI opportunity and only see a race for speed and scale. But as Scott, Mary, and Varun have illustrated today, the real winners won't just be the ones with the fastest processes, they'll be the ones with the most auditable souls. We've heard today that in high stakes environments, whether it's a surgical theatre or a global trading floor, trust is no longer a soft corporate value or a marketing slogan.

In the next five years, trust becomes a rigid mathematical requirement. If we cannot verify why a machine is recommending a life altering medical intervention or a multi-million dollar liquidation, the human at the helm will and should hesitate. The future of AI isn't about machines that act like humans, it's about machines that

It's about machines that can explain themselves to humans. If we get this right, the logic of trust becomes the foundation of everything else, giving us the confidence to innovate without losing our grip on reality.

In the next episode, we take this foundation of trust and apply it to the front lines of creativity. We'll be moving from the rigid requirements of logic to the freedom to invent, exploring how organizations can dismantle bureaucracy to let the human spirit of innovation run wild alongside these new machines. I'm Tom Parker, and this has been The Next Five podcast.