navarre.aiClarisTalk AI · the FileMaker AI podcast
Fear, Trust, and the Playbook for Successful AI Deployments
In this episode, Cris Ippolite sits down with Kate Waldhauser, Founder of Violet Beacon, to unpack why so many enterprise AI efforts stall—and what it actually takes to make them work. Together they examine how fear-driven headlines and surface-level interpretations have distorted the AI conversation, from reactions to Salesforce’s generative AI strategy to the widely cited MIT “GenAI Divide” rese
Apple PodcastsSpotifyYouTubeAll episodes
Transcript
Transcribed by machine (whisper, large model), 18,432 words. Names and product terms are sometimes misheard; the audio is the record.
All right. Well, hello, everybody. I'm Chris Ippolite, and welcome to a special edition of the podcast. I'm here at one of my favorite places, live on location in Austin, Texas. Why am I in Austin, Texas? Well, I'm here to visit with my friend and colleague, our guest today, Kate Waldhauser from the Austin area. Kate, welcome. Thank you, Chris. I'm very happy to be here. This is exciting, and I'm really excited about what we get to talk about. Yeah, exactly. And before we jump in today's topic, which I think is going to be very important for us to discuss, this is something that you and I talk about quite a bit. But first, let's back up. Kate, tell us a little bit about Violet Beacon, some of the work you're doing in the AI space, and people will naturally kind of see the types of things that we talk about together offline. So first, tell us about Violet Beacon. So I know we're going to get more into our individual stories about how we see AI, how we came to those beliefs. But really, when I formed Violet Beacon, which is about a year ago, so still a fairly new company, I became really, you know, almost infatuated with the idea of responsible AI. And so I am, Violet Beacon, what we do is we help companies really develop strategy around AI as well as implementation,
but really focused on responsible AI. And I know we'll get into more of what that looks like and what the standards are and how you develop this. But for right now, I'll just mention that concept of responsible AI. And I think that's what we're doing. I love that. I think that, you know, I think there's maybe a presumption that without deliberately mentioning responsible AI as an approach, that it might be irresponsible AI approach. And certainly there's ineffective AI, which I think is, you know, can fall into that category as well, too. But I really look forward to getting into that as well, too. And then also, a lot of the work, you know, I've been following your human intelligence work. We're going to talk about that and weave that into the discussion here today. And why is this perspective so important?
Well, I think it's because it's a very important part of our business. And I think it's a very important part of our business. And I think it's a very important part of our business. And I think it's a very important part of our business. And I think it's a very important part of our business. So today's episode is, this is a big one. And we're going to, how do we, how do these things weave together? We're talking about fear, trust, and the playbook of success for AI deployments. So fear and trust, and how do we get to successful AI deployments? Well, I think we're going to focus first on the fear part. So we want to really have a discussion. And what this is framed for is for executives who are making decisions about AI and deployments and how to measure success. And the other part that we're formally acknowledging is that this is the 2026. And really the consensus across the industry from all the pundits is this is the year that AI has to show ROI, right? And so we're going to talk about how we get there. We're literally at the second half of our podcast going to go through in a very prescriptive fashion, like these nuggets, the playbook for AI success. But let's talk about fear first, Kate. Why, I mean, why so much fear with AI? Where do we think this is coming from, right? I mean, it's certainly a cognitive bias, there's some real unknowns out there, job anxiety, there's certainly media framing, we're going to talk about that today. But I mean, what's your take on this? Why the fear?
Where's this coming from? So I like to think, you know, first of all, I read a lot. So I just am ingesting a lot of information. And that involves, right, the headlines that we're reading. But also, I personally try to dig deeper and go into and look at historical perspectives. So, you know, really curious about, you know, what are these tech cycles really looking like? And so I dig in and read deeper into these. So I've been thinking, like, one analogy that keeps coming up for me is about driving, right? It's about the invention of the car. And if you look at some of these, like, more historical perspectives of, you know, what was written about, you know, when the car came up, it was like the horseless menace, you know? And so I've been thinking about, you know, and that there was a lot, you know, they tried to outlaw cars, right? It was, this is, I'm talking about right now in, like, the late 19th century. Oh, yeah. This is when, if you saw a horse and buggy coming, you would have to move your car off to the side, get out, because the machine scared the horses. So you had to, like, get way off the road or something. It was just, it was crazy, the perspective back then. Yeah. And, you know, the streets weren't meant for cars. They were meant for people, and you could have a conversation. I mean, if you think about it,
like, the car, and I know we can probably talk about this, like, as we, you know, talk about it, but it changed our, you know, our way of living, our society. I mean, businesses were built up around it. So I think of this, like, in a way, just to remind myself that, like, this fear, right, of, oh, everything's changing, isn't, you know, it isn't new. It's actually a normal response to new technology in these technology cycles. But also, I'm reminded that, okay, and we made it through. We now have cars. Yeah. And now we're moving into self-driving cars. Oh, I took a robo-taxi in here today. So, I mean, talk about the bookend of that. I'll tell you what, we're going to do a really fun exercise. I've got a surprise for you. We're going to do a looking back in time. What technology were these headlines describing at what time period? And it's, and cars aren't one of them, but we're going to talk about a bunch of them, and we're going to test that. But first, as sort of a context, I think it's important that, I think it's actually critically important that when you're looking at a car, you're looking at a car, you're looking at a car, you're looking at a car, you're looking at a car, and right now, maybe, and I think what we're going to say, and what you just described with cars is that this is very much like every technology evolution. You start off with optimism and pessimism. And so there's a scale, there's a spectrum, so to speak, right now. And
I, and you did some really interesting work on, on your own self-assessment. So my question for you is like, where do you put yourself on that? And did you in fact put yourself there? Do you have some other mechanism that you use to determine where you're at from an AI optimism to pessimism scale? So, yeah, I think, you know, again, as I mentioned, I read a lot, I ingest a lot of information. And so I did an experiment where I took all the books, you know, that I had been reading, you know, so I'm not looking at news here, just, just books that I've been reading. And, and I, you know, put them in a database. And then I use some AI to analyze them by, by putting some descriptions of, and I chose four different categories. Now, there's many different categories you could have, but the four categories are, you know, the four categories are, you know, the four categories I chose were naysayer, doomsayer, pragmatist, and techno-optimist. So, so to define those, so the naysayer is the person who says, oh, this AI is garbage. It's never going to happen.
It's like, this doesn't work. This isn't, we're, this is going to be, you know, fly by night and it's going to be gone. Okay. The doomsayer says, you know, we're all, we're all doomed, right? This is going to destroy us. This is the end of humanity. So these are, these are the extremes, right? The pragmatist, is more, you know, of course the problem solver. It's like, okay, here's this new technology. What can I actually do with it? Like, how can we actually make it? Curious. There's a little curiosity. A little curiosity. Yeah. Not blind optimism. Yes. And then the techno-optimist is more like the extreme, but also thinking like AI is going to save us all, right? That, that, that we're not going to survive. Right. Oh, we're not going to survive. Like, well, maybe. Without it. Without it. Sorry. That's like the opposite of the doomsayer, right? Of course. The doomsayer says it's going to destroy us. The techno-optimist says, like, oh, we're not going to survive.
It's going to save us. This is going to save us. And, and so I think, you know, when I, when I took my database of books and I, you know, kind of divided it into these, or categorized it a little bit, you know, and of course I wouldn't say, I mean, these are general broad stroke categories. And of course, when you read somebody's, it's, it's more than just, you know, that, like there's a much more to the message, but in these general categories, most of the books I was thinking about that, I was like, well, maybe most, most, you know, in this, in this realm, you know, of, of writing, you know, short, sorry, long form books that maybe they're, that maybe those are more pragmatist because you have a lot more area to write. So you can write about some of the pros and cons, but some of them definitely fed into the doomsayer, naysayer, you know, techno-optimist.
So you kind of find it on all. So would you say because your information sources are pragmatist that you've put yourself in that same category? That's what I'm thinking because, because I have a theory that like, like the information we absorb and read and you know, it's, it's influencing us. Like, yes, I have ideas of my own, but really I know that all of my ideas have been influenced by what I'm ingesting, what I'm talking about with other people. Like I know that. Although I do think part of my nature is, is to be a pragmatist that, that I don't want to close myself off. It happens a lot too. And, and in our, our next segment, we're going to talk about if you just read headlines versus content that you might see something. We're going to look into a little bit of a headline Rorschach test here. So maybe you're self-identifying or pulling from something.
And I think that really speaks to what you said in the long form, you really have to explore beginning, middle, and end to your concepts. Whereas in the shortest form, which is headline reading, you can see whatever you want to see. And we'll go into a little bit of that. As a I think I would probably have to fall under optimist, but here's a really important part of my journey that I think is important to share. I started off as a pessimist or whatever the doomer was. I was lucky to get one of the first 100 API keys back in 2020. I didn't know what that meant. I was a ML guy back then. And I was just looking for another cloud resource that could do some of these cool things like classification of data, summarization, that type of thing. And so I had a think week where I was going off into some rental home in Big Sur for two weeks where I was just going to be off the grid except for a little hand crank internet. And I was going to explore this weird new API that I just got. And my initial reaction, Kate, was, oh my God, this is, whoa, this is going to change everything. I'm not ready for this. What is this going to mean for my business? I was immediate doom. So much so that I didn't even tell my coworkers at iSolutions at all about this discovery I made, where normally I would just be like, oh, this is going to change
the very next morning. I'll be like, guys, check this out. It's amazing. But here's the thing. Over those five years, I have moved further and further towards optimism for one main reason, because I read the papers, because I'm actually deploying and building in the early years, doing things wrong, figuring out what tools I had, what guardrails are available. And now we're deep into implementation cycles where we are setting expectations with clients, with ourselves, building, deploying, and actually seeing the proof points. So I'm happy to say I emerged out of doom to optimism through actual application. And I think that's probably what we would prescribe the most to people if they're curious about these things. But still, we have to interact with others.
So one of the interesting things to help us kind of get into, like, what are we looking at with these information sources, was there was an article that I got sent to twice last week from two of my biggest clients. And they're smart, really intelligent, thoughtful individuals. So I thought I would pick them and it was the same article. And this article went viral. This is the article, by the way, that was called Salesforce executives say trust in generative AI has declined. That we can kind of see what that message is saying, right? This came out in the information. It was a good article. There's a few quotes in here that really made the headlines. And I wanted to talk through a couple of these with you, because what I think this is an interesting exercise of is, well, let's look at the quote. What can we derive from this? Is there something helpful from this? And then what we'll do is apply that to some other papers and articles that we've read with the idea of giving executives a playbook for AI success. So first of all, you saw the headline first. What did you think when I was sending this over to you? Well, I think it's, it's, you know, I really try not to not like to take a headline with a grain of salt, because it's, it's not telling, right? So I think I know. I'm
like, oh, okay. So, you know, just to, just to, you know, zoom out for one second, right? It's, in case it's not known by, you know, most people that, that headlines are meant to sell a story, right? It's like, they meant, they need you to engage us, right? This is, and that's obvious. Like, it's obvious to say it, but I think like psychologically, we, we might forget that, you know, and, and there's so many, I think I looked up some number, there's like over 200 cognitive biases, right? So, so I have a little bit of a background in psychological research, and, and what it, what it shows you is like, we're, right? So, so by reading this one sentence, five words, however long the, the headline is, we're going to have an immediate reaction, and, and those cognitive biases are going to play into it. So, so that bias says something that agrees with what, like, I already believe, then I'm gonna be like, oh, it's confirmation, right? See, there it is, right? But it really takes, like, yeah, you kind of have to dig a little deeper. So, so back to that. Or read beyond the you know, passing judgment. Right. I do this probably every time I scroll through my timeline.
I'm not pretending that I'm, you know, you know, you know, infallible here. I get it. Yeah. I mean, I do it too. Like, sometimes, like, I don't have time to read a whole story. I'm just gonna read a headline. Like, I can't, I can't make time for every single thing. But that article, you know, yes, I, I looked at it, then I read it, you know, of course, because I was like, okay, well, what does this mean? So, I guess my initial reaction is really curiosity, is why did they say that? And then I want to look into it and understand, well, what, what made them say that? So, yeah. Well, I think that, that's wise. I can't say I always do that. But in this case, I had to, because I actually got sent this from two customers, two of my clients, and they were asking for my opinions. So, of course, this was a drop everything, look into this and see what this actually means. In one case, it was a front office client that's actually, you know, big Salesforce user, and they were looking at it more from the Salesforce perspective than the AI perspective. And in another one, we were actually building, you know, we're building something that relies heavily on the AI perspective. And in another one, we were actually building something that relies heavily on AI. And we're wondering, hey, is this going to be the best bet? So, and actually, you know what, there's definitely a bias in media. You know, anecdotally, I can share with you that a couple of years ago, one of my PR friends asked me, hey, you should go on one of these shows and, you know, respond to some of these anti-AI narratives. And I'm like,
yeah, I don't, you know, it wasn't a good place to be professionally a couple of years back, like being the one person saying that AI is good. But I said, where are you getting these from? And he's like, you know, if you're writing an article and you're a reporter or something, or you're doing something on television or one of the networks, you can put, you know, go to these, I don't know, places where PR agents or experts, you know, people will look and you'll say, this is the article I'm writing. Here's what my intent is. This is what my approach is. And so I'm looking for someone that can speak to blah, blah, blah. So I asked him if he could download all these to me from his system, which he did. And I ran through, I don't know, there was like 1500 of them from like a few years. And I found 83% of them were from AI. And I was like, oh, that's a lot of people. And I of them slanted negative against AI. So I, that was how I determined, well, I don't want to be that guy, the one guy who's getting beat up. Cause I don't think they're looking for someone to come up with a, you know, a constructive response. I think they're looking for somebody to beat up on. So, um, so I was wondering, is that the case here? Well, there's a couple of things that really stuck out here. This, by the way, this article went viral. Um, a lot of people picked up on it and it was really because of these four quotes. And I thought maybe you and I could dig in on the four quotes, because I really think that there's really positive, very positive things to take away from these. Um, that might be my optimism, but you know, maybe you can bring a
little bit, uh, you know, further towards the center, uh, perspective of it. But the first thing is, is the use of the term determinism and actually, um, determinism or deterministic has become a pejorative term in the AI world, which I find fascinating because a little bit of background, you and I come from the most deterministic world in tech that we can, and we're in custom software. You know, this is, uh, you know, we're in a, an industry that's been around for 40 years and we've always built totally deterministic predefined workflow applications so that when our users run these things, they always go boom, boom, boom, boom, boom, all the way through the process. So first of all, determinism is a bad, right? I mean, well, how did it get so bad?
Yeah. Yeah. Yeah. No, I, I, I do completely agree with you. And, you know, I think, um, because I remember watching, um, um, um, the Mark Benioff when he, when he made this big announcement. Well, he was like, so just to frame it up. So, uh, this was in 2025, maybe end of summer. Uh, oh no, it was the fall. It was probably at Salesforce. I think it was December 24. I don't know. Yeah. Okay. Oh yeah. Oh no. Right. When he said they were going to move towards agents, they're going all in on agents. I'm sorry. Yeah. No, even better. Yeah. Because this article is really about agent force. It is. And, and so, so he made some very big claims about like, yeah, we're, these is good, you know, and, and there were a lot of claims made about all in agents. And so when I read that and the determinism, I mean, first of all, I guess it makes me feel a little bit better because as we know in custom software development, things don't always go exactly as you think they're going to go. Right. That, that sometimes you have to pivot and that's actually a really good thing, right? You need to be able to pivot and notice, okay, this isn't working. Let's go in this direction. And you need to be able to, because if you, if you knew everything ahead of time before you announce it, like, you know, now this is the CEO of Salesforce. I know this is a little bit of a different deal. Well, and Mark Benioff is known for being a bit of a
Barnum and Bailey since, I mean, he's very effective. He's the no software guy created SaaS in the first place. I mean, it's brilliant to, you know, he's historic in this. So he did it there though with agent force. Right. So I think when I watched his, I probably took it as I take a headline, which is, okay, let's, let's see how this turns out. Right. Cause he's making claims, but they're futuristic. Like, how do you really know? How this is going to turn out. But the decision to bring determinism back into the software or, or in whatever element makes perfect sense. And like, what I think a lot of people don't realize is this is happening. Like, I mean, AI in itself is a little bit hard to grasp at times, right?
Because, because it means so many things and it's a huge, you know, but determinism is, is brought into, is into the process of, of making AI, of training AI. Right. So, so that's how they put I think, you know, from a business perspective, determinism is a very valuable tool and, and it's, it's actually in AI to, to a degree more than we think, you know, this concept that, you know, a model is doing everything and it's, and it's, you know. Well, look at the system prompt. You can, you can just go search for system prompts for all of the consumer ID products out there. You can find the system prompt for CHPT, clot, all of them. Right. They are literally are starting every interaction that a user has with this deterministic construct.
Right. They're instructions really, but they're intended not to just have this wild west experience with the models. Right. And, and I think you, determinism is being kind of, you know, while it was, you know, being deliberate and, and, and having a structure to what you're trying to accomplish with technology in the past, now in the AI era, it's almost become the antithesis of autonomy. And I think what, if anything, if you can see in some of these things that are in this article and Salesforce's position, by the way, they dog fooded, Benioff. Dog fooded his, his approach. He, they let go like 200, 300 hires or went, did not hire more to fill these positions because they were using agents, agent force specifically internally as well too. But from my experience, what I find is that, you know, 2025 was the year of the agent. And first of all, there's a million different definitions of what agents truly are. But for our conversation today, let's frame agents up as allowing the LLM to create its own plan, introducing it to the two tools and potentially data sources that it wants to use to be able to, to answer the user's question, let's say. And then at the end, it comes back, back and it uses the reflective properties that we saw emerge in the GPT-4, 4.5 era of reflection to be able to look back on its plan to determine
whether or not it executed properly. Right? So an agent can do all of those things. And so the promise, especially in 2025 was like, oh, you can do entire job roles and just let the agent do all of those things, let this thing run, do its thing autonomously, and just YOLO AI against the job. Now, if you look at that again, and versus determinism mean that everything is totally structured, they're not black or white. It's not one or the other. But if you look at what their promises were, they did say that they're executing a 93% of it within agent force. They're having 93% success. I, I can't even believe it. I mean, that's it. And I'll take it all day long, 93%.
But what they're saying is in that, that Delta, in the 7%, they introduced deterministic guardrails, meaning more instruction, more sort of, you know, controlling the behavior, which I imagine, what's your take on this? I would assume it had to do with what the actual agent use case is. Like some probably require more determinism than others. I would think more than 7% require determinism. Yeah. Yeah. And, and it, like you said, like there's nothing, there's nothing wrong with that. I mean, I don't, I think this conception is a misconception that AI, that this non-deterministic agentic AI is supposed to do everything that, that, yeah, we're supposed to be. And I don't believe that, by the way, maybe that doesn't make me a total optimist. I don't believe that agents can do everything. Maybe we're not there yet. Maybe we'll get there. I don't think we have to though. It's a very hammer and nail in our industry with agents everywhere, really. Yeah. I mean, you know, it's sometimes, you know, I think about the future.
It's like, well, we, we just have no idea what's going to be like in, we're saying this now, 20 years from now. I don't know, you know, but. 20? Two? Hey, hold on. 20 years. No way, man. 20 years is way too hard to imagine. Oh, yes. But I was thinking that the, oh God, I just lost my train of thought. Well, I think. I was thinking about my daughter in 20 years. I was like, oh wait, she's going to be 27. What is that? Like, I don't know, my brain went off. I don't even, I can't even squint that hard to see down the road. I mean, two years seems impossible. But what I can tell you in the last year is that determinism is part of the process. And we're going to talk a little bit about picking the right place to integrate AI. But what I've seen, and I can be very prescriptive with this, is that you should start with trying to build a base where your goal is that AI could autonomously or dynamically adapt to whatever the inputs are, but you should immediately move into providing the guardrails to kind of force it in the path that you want it to be. Not like, like putting in, you know, if this, then that type approaches, but literally just saying, hey, look, there's a place for when we want the nuance that a language model brings. And then frankly, there's other places where it's just not necessary. Some of the things in this report were pointing to using agents to create a report and guide somebody through filling out the, you
know, form. And it was going off track sometimes. And to me, it's like, why are we using AI to fill out a form? Like, I think we solved that problem already, for example. So that's sort of the thing that I think about, and specifically some of the things they were pointing to in this paper. So I think the takeaway, I think we both agree here, determinism, start with determinism, perhaps. Start with a framework that can answer things as dynamically as you're able to execute on, but then start with deterministic use cases and then branch out to more general ones from there. But the key is don't start with deterministic programming and try to branch out from there.
Because what I've experienced, and I've seen this firsthand, is you'll just paint yourself into a corner that you can't get out of, and you'll end up having to start all over again, which I think is one of the reasons why these AI pilots are failing. Which I will actually see in the MIT report. Let me ask you this one, though. Prompting. A lot of people think interacting with AIs is just everybody has to be a prompt engineer. You have to have this big, long prompt. The term for this is called single-shot prompting. I do use that term pejoratively when I identify maybe some single-shot prompting that I'm seeing happening. Well, that wasn't directly addressed here. So let me get your thoughts on this quote. So one of the things that I found interesting in your response here, so if you give an LLM more than eight instructions, which, by the way, sounds like a lot of instructions for one prompt. So I think what they're really saying here, if you give a single prompt to an LLM more than eight instructions, it starts dropping them. Combine that with another quote from folks at Salesforce that says, agents will lose focus when users ask tangential questions. By the way, that second one framed up the agents guiding someone through creating a form example I just mentioned to you a moment ago.
You're prompting exercises. Don't you sort of get like that little itch, like I'm giving it way too many things for one API call to an LLM in the first place? Would you expect eight instructions to work in one call? Well, I think this is where the AI fluency, there's a gap here because if you've never used this tool before, you don't know. There's no instruction. I mean, yes, there are all sorts of people telling you how to use it. But it's an empty chat box, right? It's just a blank slate, right? And you can put whatever you want in there. I mean, then you'll start to hit the limits, right? As you realize, oh, wait. You're talking about literally context. But this is something that you learn as you work with the tool, right? I really think that that's where the best learning comes from. It's not something somebody can actually give you a lesson in. It's something that you have to learn and play with it yourself. So if you didn't know that, then you might expect, oh, this is going to work this way. And I think that one of the other things that I'm really focused on or thinking about lately is the model isn't meant to do everything, right? I think
there's this misconception that, OK, you're going to have this model and it's just going to do everything. Really, what the skill is and what we're working in is what you and I have been doing. And so I think it's really important to think about that. And I think that's why I think we're doing this. And I think it's really important to think about that. And I think that's why I think it's really important to think about that. And I think it's really important to think about that. So taking just ChatGPT out of the box is useful, but not for everything, right? There's a way, you really need to look at it and finesse it and look at your use cases. So I think that plays into this in that prompting isn't everything. You're nailing it. I mean, what I can say practically, I'll give a pro tip here. First off, language, what I've learned is that language is I like to come at it from this. This is where I've seen the most success for our deployments.
Instead of having the LLM do everything, assuming that it's this PhD level, all-knowing being that you're going to have to take on all these tasks and all the decisions that you need in your workflow, I look at it at the opposite. I want to introduce as many control mechanisms like the term context engineering became really popular last fall, but it ended up that it's doing. Leveraging techniques like RAG, retrieval augmented generation, to bring in your own truth contextually within your interactions, or using stateful memory to actually introduce persistence across these workflows, or just giving it tools, tools like people who are doing it right know that LLMs can't do math, for example. So instead, identify where math needs to happen, hand it off to a Python environment, or to do sophisticated execution using code execution. Once you realize that those tools are available to you, and they're all totally deterministic, then you can then say, well, you should look at your entire workflow and then say, all right, well, do I need an LLM for this? And maybe do experiments to determine whether or not you actually need it. What I found coming out the other end of the tunnel on these many times is you really don't. If you just reduce the
load on the LLM to just where the LLM's nuance really comes into play, you can really get some outcomes that way. So when I see something like give an LLM more than eight instructions, to me, I'm like, your workflow should have as many independent calls to an LLM as you can. You should endeavor to make as many independent calls as you can, mostly because then you can have reaction to them. You can determine whether or not those are tool calls and different things are leveraged that way. So if you're trying to, if you find yourself, not you, just the royal you out there, if you find yourself trying to force everything and you start putting conditions into your prompt and you're saying, okay, and if it comes back and does this, then do this, hey, you just identified a turn. That's where you should actually break these up into two different ones and then really look at whether or not the different calls need to actually be to a model or a tool. And I think then determinism becomes positive. And that's what, that's how we build applications. We've been doing this this whole time, right? So it's not that unusual. Right, right. It's, it's, I think in a way, AI is just a natural, you know, progression of software development. You know, I mean, and, you know, we haven't even talked about, you know, vision models and robotics. And of course that plays in all this. We're focused more on like the language models, which of course
is, is, you know, what's more, you know, but, but these things are out there too. And those are incorporated in business as well. You bring up a really important point. Another pro tip I would throw out there is right model for the right job. If you're doing it right, you should not have one big monolithic prompt to speak into AI and expecting it to do a bunch of things. Let's say you shouldn't expect a call to an LM to do more than one thing. How's that? And if you find it trying to do eight, break that up into eight things, right? The other part is it's not all the same model. I, the, the, the, the experts out there, the really good heavy lifting AI developers out there that are really moving the needle. What they're really good at is making the smallest possible model do something valuable.
So we're going to see a rise of SLMs. It's going to get really awkward when everyone calls AI LLMs because we're in the era of SLMs, small language models, or moving that inference to like the edge or that reasoning to the edge. So I think the more you can get done, first of all, the first pro tip is break things up as much as you can. The second pro tip is don't try to use the biggest, smartest model. As a matter of fact, you have to be cost effective too. So use the free ones or the cheap ones first. I think it'll shock people when they see how capable some of these SLMs are and these, these on-device models even are. And then when you really need the heavy lifting, push that out to a lot of people. And then when you really need language model. So eight instructions is too many. It should be eight calls. And then agent lose focus when users ask tangential questions. Well, capture those interactions so you have control over them. And you do that by having multiple calls. Sounds a lot like custom application development, doesn't it? We didn't have one script that runs our entire application. We had them embedded into the workflows doing one thing or a couple of things at most, right? Right. And I don't know. I mean, sometimes I think it's because because LLMs get so closely related to the human brain, right? I think that's what's happening. There's this it's called brain computational
theory in our computer. I don't know, something like that. I'll have to. But anyway, there's this idea that like, yeah, I mean, it's just an input and an output right in the brain. Right. I don't really believe I think there's a little bit more to that. But like the same with LLMs, it's input and output. And if we it's like, well, I use my own brain to do many different things. Right. Is it this conception that, OK, well, then I just need one model because it's like a brain. And I and I and but but I don't think that's true because, OK, language is only 10 percent of the brain. Right. So so all of my brain power is being used in all of these other ways and senses. I mean, it's it's way more. Right. So what we've built with a neural network, with LLM is only an aspect of the brain. And, you know, there's there's there's some great labs and people doing work like Fei-Fei Li in World Labs. Yeah.
It totally what I was going to say. You're nailing it because there's a theory. There's people out there and don't don't get them wrong when they like Fei-Fei. If you ask her, she would say, well, we cannot achieve. I think she's of the school of we cannot achieve AGI or super intelligence through language models alone. I mean, we know some of the biggest ones, the biggest minds out there say that, but they're not saying that I can't do it. They're saying that we need world just to your point. If 10 percent of our brains are our language. Well, Fei-Fei is actually building worlds, literally worlds. By the way, if you don't yet believe in simulation theory, you will after you see the work that they're doing over there. It's insanity. But let's just pretend that I don't for a second. But what she's saying is just like we need visual inputs and audio inputs and we need a model to have all of these things to truly be able to be responsive in a general fashion. So just to your point, the brain works that way. So those people who are advocating for that.
They're not saying language models are not useful. They're saying isolate them to language tasks. Yes. Yeah, exactly. And we need to remember that because it's like people think AI, they think chat GPT. Yeah, it's not. This is not. It's it's it's a it's a very minified view of. No doubt. So I mean, probably the other thing that's so interesting about people going right to chat GPT, Kate, is like how this all happened. I mean, for the first couple of years of working with AI, I found myself having to explain to people what I was even talking about. And then that fateful November 2022 chat GPT comes out and, you know, the fastest application to a million users, active users in the history of man. I mean, it beat out Instagram and all the most viral apps of all time. It became chat GPT. It became the Band-Aid or the Xerox or the whatever of AI. And it's crazy to think how like there were no other players at the beginning of 2023.
And here we are three years later. And now there's a ton of players here. So yeah, it's fine. But also it made it so that my father knows what AI is and people who aren't interested in technology know what AI could be. But it goes beyond chat. And so really, to be careful, we're not talking about chat being the only thing you can do, chatting against data or whatever. We're talking about implementing AI subtly into your workflows here. Right. Right. But I think in that Salesforce article, they really are referring to LLMs. Oh, 100%. The gigantic AI that they're discussing. They absolutely are. Yeah. And the key there is agents from a technical perspective are LLMs with memory and tool use, right? So it's not other models, like vision models.
Unless the tool use could be, you know, so there's, right. And then, yeah. Great call. Well, that's how your mind should work. When you're looking at these things, you should think right model, right job, right tool, right job. And when you start looking at it that way, you're going to not have a lot that the LLMs need to do is what the thing, is. Yeah. Here's my favorite one. You know, you and I come from application development world. One of the quotes in here says, better observability lets us see when agents go off topic and iterate quickly. That's kind of like the no kidding Sherlock type thing, right? Like, oh, you mean test? Right. You mean actually, like, so for those that aren't familiar, if you're embarking on your first AI application, or you're an executive and at a company deciding, you know, what, what your team construct is going to look like, make sure that having an eval framework in place is step number one. You cannot start building and we're not in, and just to clarify, uh, for ML ops folks that have yet to embark into the gen AI world, this is not benchmarking. This is not QA benchmarking this. It goes beyond that because evals are about this black box that we're using. How do we determine whether or not it's being accurate with certain question answer pairs that we're giving it.
But the next step is, Well, if it wasn't, it didn't meet our acceptance criteria. Now, what do I do? So what this gives you is observability into what little knobs you can control. When I give more context to this, does it, does it, does it give me more accuracy or whatever it is? You can measure relevance of responses this way too. So eval frameworks, critical pro tip before you get started. It is amazing to me. I mean, I guess Salesforce is saying, yeah. And then when you actually use frameworks to test these things, you have more control. Um, I know your thoughts. I mean, I mean, in fairness, there's a lot of developers who don't test present company excluded, but I guess this is a human nature thing, but I don't know. It's not that odd, I guess in that regard.
Like going back to my, you know, car analogy here, you know, I think about like, okay, if I was comparing this to, to the car. So, so what were, what were, what was car development like in the, in the 1890s? Right. I mean, it was very rudimentary. It wasn't until 1913 that the, um, the assembly line, right. The Model T right. And so I'm like, okay, so there there's this time. And I think we're operating on a much tighter time scale than we were at the turn of the century, but that there's, we're still learning and understanding best practices. And I think that's where evals fit in. It's, it should be obvious, but because this new technology is here and we're like, kind of like, whoa, what do we do? And, you know, some people relate it with like magic, you know, and there's this thing. And so we have to kind of rein ourselves in. And I think this is where having a background in, in the custom software again, really helps because we have the baseline and the tools for all this and.
You know, in Responsible AI, which, you know, I, I'm going to bring in here because in Responsible AI, it's really just about a set of principles, right. That, that we're using to guide, um, to guide, you know, every step of, you know, every step of the, the model or the, you know, AI life cycle. And, and one of those principles is transparency. And that's actually, you know, observability, transparency, interpretability, all mean slightly different things, but they're all rooted in the same, the same idea, which is, you know, you need an audit log. You need to be able to see what actually happened, right. That's the traceability that they're talking about here. Yeah. You need to go back and, and look at like, well, why was this decision made? Right. And that's the interpretability, which, and yeah, people talk about the black box, you know, aspect of, of these models, which again, there's new research coming out then and new ways that they're actually beginning to trace how the decisions are made in the black box.
So again, like we're still in the early days, like we haven't reached the assembly line yet. I don't think. I love that. We haven't reached the assembly line yet. Yeah. But we have brakes on the car. Right. We can still get from A to B. Right. And, and, you know, and it took those things and, and, you know, I love that. Well, I'll get, there was, gosh, I can't remember his name. Eno. Eno. He was, he was, um, he was one of, he was an individual and he was really fascinated with, he, he got in his first traffic jam when there were still horse drawn carriages and he was like, why are we in this traffic jam? So he took it upon himself before the government started doing this to build the first traffic circle and to build the first stop sign. And like, these are things, and so these were developed from private, right? From, so I think this is where it's like, we all have a part in this. Yes, the government is doing things and yes, there are global standards being developed, but, but it's actually up to the individual.
And there's nothing stopping us. And it makes just good business sense, as we're learning from Salesforce, right? It's like, you just need this. It's not, it has to be a given. It can't be an add-on. So as your pro tip, if you're an executive deciding on team building, uh, whether you're going in-house, out-house, out of house to, to form a team, think about these eval frameworks and think about the, you know, responsible elements and how you're going to control those right out of the gate. Is that a fair way to look at this? Yes, I think so. I think they're, they, just have to be, you know, a must have if you're developing an AI system that, that you need an eval, a way to test it. Yeah. It's, well, you're not going to get far, honestly, you know, for, for organizations that I watch kind of learning on their own, um, that might not be as open to a suggestion or influence. I kind of sit back and go, Oh, you'll get to the need for evals very quickly. Whenever you're done with your big, long monolithic prompt, and you try to YOLO that against the biggest language model you can find, you're going to find out that you have a lot of areas that you need to, um, isolate, uh, and to be able to improve upon. And it's going to be great. Those are going to be the tools and the knobs and the dials you need to get to where you're going.
Um, so on our article thing, our headlines from reality. So, um, you know, we talked about how some people can look at the headline from the Salesforce article and say, and think a certain thing right away. Um, I like to think of this a little bit as, uh, the, the AI headline Rorschach test. Okay. Now, before we get there, I just want to say you sent me yesterday as we were preparing for this, a tool, a GPT that you created. You want to talk a little bit about what you did there? I thought it was great. Yeah. You know, so, so this is the really cool thing, right? And I'll talk about why I think this is the, this is one way to use AI, right? Is you can build a custom GPT, right? So that's what I, that's what I built. And, and I put in some, um, some instructions and I basically wanted to be able to upload either an article or a link to an article.
Yeah. And have AI, you know, read the article, kind of analyze, um, analyze what was in there and tell me, you know, was it, give it, I gave it a grading rubric, right? So a way to score it and then tell me what level it, what, what, where it landed, you know, was it, was it signal? Was there actual truth in here? Was it hype? You know, was it like, Hey, there's no real, um. Sensationalism or something. Yes. And so, um, I built this tool and, and I haven't really evaluated it totally yet, you know, but it's like, this is a proof of concept, right? I can get it. Pre-eval. Yeah, it's pre-eval, but I could build it really quickly and then I could start like testing it, you know, and, and then giving my own, you know, and reading my own, but it, because, right, we're dealing with language, right? And so much of these articles are just the words that they're used, like, right, how they're used. But then I could also program other, you know, elements in like, you know, are there supporting facts in here? And, and what was the incentive? Like who wrote the paper? What were they getting paid to write it? Right. Because a lot of that, you know, we, we forget about that.
With like, well, what's the incentive of the person who's, who's creating this? So. Well, the media's job is to be centric, you know, center and to report objectively on everything. I mean, we know that. I mean, I'm, that's the sarcasm coming out in me right now. As a matter of fact, I'm going to play a fun game. We've been a little dry here for folks that have hung in here with us. I'm going to have you be the human headline evaluator tool instead of AI. This is going to be fun. These are real headlines from, the emerging technologies over time. I'm going to all the way back to the 1700s here, by the way, because to your point, when you look back on like the beginning of, you know, 1900s with cars and you look back on it now, you kind of chuckle to yourself because you're like, oh, how silly, how short my sighted we were. Well, we weren't less intelligent back then. We were, we just didn't know what the future holds yet. Right. So to prove that the initial reaction from humans to new technologies, so to speak, can skew.
Negative. And then we eventually settle into it. Let's look back in history at some evolving technologies. I've got about five quote unquote technologies. Now, of course, we're talking about the 1700s. We're not talking about anything you can plug in. So you got to think, you know, a little bit here. So I'm going to read you the headline and give you a couple of the quotes from it. You guess which thing they're talking about. Okay. So this thing, comma, a cause of female depravity. So from 19, for, from 1797 in the evangelical magazine, they was, this was talking about this thing was being blamed for corrupting the morals and unhinging the mind of young women and saying that young women could become unfit for domestic life and incapable of reason if they engaged in this new emerging thing in the 1700s. What might that thing be? There's no, I mean, this isn't a test for you. Right. I know. Now you're curious, right? Yeah. Now I'm, I'm stumped on this one. Well, it's the novel.
So literally, I was going to say reading. It is reading. It literally says the headline says novel reading, a cause for female depravity. Yeah. Yeah. You know, uh, unfit for domestic life and incapable of reason. I think that's proven to be false as time came by. So again, we look back on it and it's, it's literally laughable to think about it that way, but hopefully we can gain a little bit of context here. Okay. Next one. Are you ready? Now we're going to move into the 1877 from the New York times headline, the blank unmasked. This device, was accused of enabling constant unseen eavesdropping. There's your cue. And it was warned that it would destroy all privacy and fundamentally alter social relations. That's unfamiliar. And I've also heard it was used to communicate with the dead telephone. It's the telephone, the telephone unmasked from the New York times, 1877. Well, um, the, the communicate with the dead thing, maybe that never materialized, but the rest of it, but some of these are kind of interesting, destroy all privacy and fundamentally alter social relations. Interesting.
Right. Right. Um, now we're going to get into some real doomer stuff here. Kate, here we go from 1889, the New York times headlines, the following blank as an agent of death. Oh, this thing was described as a mysterious force, which kills without warning and it could strike invisibly and fatally at a distance. And this is 1889, 1889. There were, uh, let's just put it this, there was a war going on over two different types of technology. This is the bicycle. Oh, no, no. Um, think I wrote a robo taxi in this morning from the company Tesla. This was electricity. So the old electricity battles with Tesla and Edison, right? Right. So they were saying electricity is an agent of death. Just to revisit this. It's a mysterious force was with, which kills without warning. No question. Get rid of it. But we learned how to safeguard it. We learned how to literally have babies in a house with electricity, you know, electrical outlets that will not, you know, kill without warning. So we learned how to safeguard it. We learned how to, you know, kill them without warning. Right. I mean, not to get too crazy there, but this was seen as, oh my God, this is going to kill us all because it's just too powerful. And then we figured out how to harness this. And now literally we are surrounded by electricity and neither of you, you or I are concerned about that. Um, last two lunacy in England, 1894 from the New York
times. There's not the slightest doubt that this activity if persisted in leads to weakness of mind, sounds familiar, general lunacy and homicidal mania. These individuals, individuals who partook in this new thing, uh, uh, become a prey to a thirst for homicide. What year was this? 1894. I don't know. This one's got me stumped. Bicycles. Oh, there's the bicycle. So what was the, you said you, you picked up an anecdote about bicycles before, uh, or you were, that was your guess. That was my guess. I guessed because it was around the same. Yeah. It's around the same time. Interesting. But they said strike from afar something. Oh, maybe a bicycle is flying down the street. I don't know. Sure. Who knows? But I mean, obvious fear, right? These are literally headlines. Final one. This is, we're going to come into a little bit more modern age, 1954. You can probably figure this one out. The headline from the New York times at the time, the, a new menace to the minds of the young. This was said to cause passivity, intellectual sloth and emotional disturbance in children. Also the said it would short circuit attention spans and weaken the ability to think critically. By the way, I just headlines right now describing AI in that manner for children with developing minds. What is this thing from 1954? It's talking about the TV, the new menace of the minds of the young. Now I
don't, maybe that one, maybe that one we don't disagree with too much. Maybe we're not a hundred plus years separated from it, but, but interesting though, at the, at the very least. I know. Well, yeah. I mean, and in one way, you know, it's like they're like electricity that, yeah, they're, they're very dangerous. There can be, you know, don't, don't put your finger in the socket, right? There are, there are real, there, there's, there is some reason behind it. So I don't want to like, and the reading the novel, that doesn't seem to be, well, sure, but they can't all be fantastical. Yes. With every new technology, yeah, there is some valid fear or danger, but that's why we created laws around. That's why we wear seatbelts. That's why we, we, we create all of this around it. But if we were to turn away and ignore the technology, it would be, I would just like to turn away from novels, telephones, electricity, bicycles, and television. Well, some beget the others, but it would, we wouldn't be here right now. You and I wouldn't. I was just reading. So the book I'm reading right now is Super Agency by Reid Hoffman and his book. And he, he tells this story. Reid Hoffman, LinkedIn Reid Hoffman. LinkedIn Reid Hoffman. Very involved in the AI debate. Yes. Yeah. He has a couple of podcasts and he's talking about it a lot. That one didn't
show up as optimist on your book evaluation, I bet. No, it did. Okay. He's a techno optimist. Yes. So, so, but he, he paints this really interesting picture. So he tells the story of the Luddites, you know, of course, like people familiar with it. Well, people hear about Luddite, but they don't know it's the real Luddites. Talk about that. Right, right. So the Luddites were, you know, a group of people in England. This is around the, the industrial revolution, right? They, they saw the, the new machinery coming in, right? And it was the machinery that was going to take over their jobs and the weaving, right? So this had to do electric looms were some of the first, uh, to emerge in the industrial revolution.
Exactly. And so, so there was a man, I can't remember his first name, but his last name was Ludd, right? And he was kind of the leader of this, of this group of revolutionaries who just said, no, we're not gonna, you know, have our jobs taken from us. And, and they started, you know, forming these, uh, groups in the midnight, they would go and they would burn down factories or, you know, demolish, you know, and I think they even, they even murdered a factory owner. Like, if you look in our history, it actually gets quite violent, but regardless of that, these things are happening now. Yeah. Like reactions to technology, we're burning things now. I mean, it's literally happening in present day. It's crazy. Right. And that's why it's important to look at the history as we've been doing here, because like, honestly, like we, we haven't, like we've changed and we modernize, but we go through these same cycles and we, there's the same reactions. Well, just last year, there was a very public, uh, news cycle item of a, a dock workers union that was, that was renegotiated. It was really, they were going on strike is why, why it made all the headlines because they said, well, we want to renegotiate with, uh, with our employers, where the union wants to renegotiate their terms because the docks were starting to become fully automated, like fully automated. And they were essentially saying, we need to keep having humans do these jobs, whether they're less efficient or not,
that might be the perfect facsimile of the Luddite industrial revolution. And that were, because how long can that persist before eventually just efficiencies and scale come into play and those decisions become steamrollingly obvious. Right. So, um, and in the story, you know, in the actual story of the Luddites, right. The English government eventually, you know, you know, tamped down on this and they, they, well, in trade schools emerged and literally deliberate activity, like literally the trade school concept came out of that time period because they said, well, we do, we do have to take a responsibility as a society as, how do we retrain folks? We can't just say, good luck. You used to, you know, weave, uh, you know, textiles. Now you're just out of a job. No, let's figure out what else to do. And, and, you know, people often talk about how it's like 65% of the jobs, uh, today did not exist at the time of the, at this time, this period of time that we're talking about. So, you know, we were promising a history or futures of abundance, but I think also the idea of new, you know, careers and opportunities certainly can emerge as these technologies. Exactly. And so, so in the, in the book though, Reed, he, he, he puts this little fictitious, he's like, imagine that the English
decided that the Luddites won, the Luddites won, right. So the British government decided, okay, we're going to enact a law that says we're not going to, um, and you know, we're going to prohibit any technology unless it has been proven that it's a hundred percent absolutely safe. Right. So, and then he, he, you know, this fictitious idea of what the world would have looked like and the picture he paints, of course it's fictitious. We don't really know, but that the rest of the world, right. It's like technology goes on the way it did. And there's an invention of cars and trains and, um, you know, and TV and everything. And that the, the England, you know, keeps, Oh, that new technology. No, no, no electricity. No, we got to get rid of that. We got to do that. So what happens is all these, you know, they're making these beautiful blankets and they're beautiful. They're more like the Amish, right. And nothing wrong with the Amish. Like, I think that's wonderful and beautiful, but I cannot imagine our whole world being like the Amish and what that, what, what that would actually do. Like I'm actually very comfortable in my modern life. And I, because of the modern life that we live in, it's, it's due to technology and innovation. It's such an interesting thought experiment, Kate. I mean, I really think if you're very pessimistic, present company excluded, of course, you should really look at your environment,
look at these creature comforts that you've gotten so used to. And I think you'll find that AI has already bled into some of the things that you rely on and, and think are useful. So love that, love the Luddites. I know that's a pejorative term as well these days, but really looking at where that came from and knowing that there's, and that thought experiments is really kind of interesting. That's a fun journey. Now, from an AI perspective, that same part of the world is probably the most like resistant to AI from a legislative standpoint, anywhere in the world. Um, you know, sans, uh, save for China who they, they really just care about what's happening internally, not they're outward facing the world. But if you look at early AI restrictive legislation, not even responsible AI, this is a different thing. This is just restrictions on AI and some of the attempts that were happening here in the United States. What ended up happening is, um, can you name all of the AI model creators out of Europe right now? Well, we've got Mistral.
That's it. That's your, I mean, I'm not, I've saved for a few, but Mistral, Mistral or Mistral, Mistral, I believe is what the correct term is. Like you said, um, they're the ones and they emerged out of France before these restrictions. And so have you seen a proliferation of new model builders? No, you haven't seen that. So it's an interesting experiment just in the short term, even when you look at restrictions that way. So. Yeah. And, and, you know, the EU AI act, you know, I know that it's, you know, when it's a few years because before it goes into full effect, but I was just looking at something the other day that they've, they've released this new guideline, which is, so here's the law. And then like the, the framework is like the guidelines is like how to apply. And the EU AI act really applies to the foundation models more so than anybody. Right. And so the, the companies that have signed the list to, um, to, you know, comply with these, with these, you know, frameworks and, and before, you know, it's time is ready. I mean, Google's on that list, open AI is on that list.
So all the big ones are complying. And I think that just to be clear though, they're not changing their California based models to worldwide to fit these restrictions. They're going to have different restrictive models in deployed in those, in those places. So I think it's, I think it's actually Reed's experiment that will play out if that ends up being the case. I mean, we don't even have to speculate. We'll just kind of wait and see what happens. Right. Um, but again, I'm not prescribing full on YOLO, you know, uh, risk be damned, um, which as a matter of fact, uh, what we really want to leave people with here today, uh, is prescriptive advice as to how to be successful with AI pilots. And we talk about pilots. Um, I think one of the most useful, uh, papers of the year came out towards the end of last year.
It was called the MIT NANDA, Gen AI Divide is what they were framing up. There's called this, it was one of their state of the AI reports. A lot of organizations do this. Whenever the MIT one comes out, I love Gen AI Divide. It's called the, it's called the, it's called the, it's called the jump and all over it. Um, so before we get into what the report actually says, because it actually had great nuggets, uh, you can say, well, these are why some things fail, but this is also telling us why some things don't fail. So what we're going to do is we're going to leave decision makers of where the rest of our conversation here today, we're going to focus on how to do the things.
So your pilots don't fail and you can move them into production. But before we get to that, just to touch on the headlines one more time, when the MIT NANDA, you and I talked about this on, and so, uh, I know you're chuckling because you don't where I'm going with this, but, uh, when this report came out and you can check it out yourself, draw your own conclusions. These are, if you do a web search right now and you put in the name of the paper, these are the top five headlines. Okay. From Forbes, MIT finds 95% of Gen AI fail, uh, Gen AI pilots fail because companies avoid friction, whatever that means. AOL, which is a news source, um, says MIT report, 95% of generative AI pilot at companies are failing. That's it. Nothing else. Entrepreneur magazine. Most companies saw zero return on AI investments says study. And then finally, I actually found one in the top five from a source called the stack. It said, this is what the tone of the rest of our conversation is going to be. What CIOs can learn from MIT's AI report beyond the doomsday overhype. So they just straight up went at it. So the point is like deep, deep, deep in this paper, it said, 95% of the, uh, the, the, the pilots that, that they, uh, addressed in this survey of actual
business users and executives, uh, a pretty large swath of, of, uh, uh, survey here said that they failed. And then they, the reason was why are they failing and why, and how can they succeed? So if you just stop it, 95% fail, you're not going to learn anything from this paper, right? So, I mean, your thoughts on that, um, it's like the perfect headline. Yeah. I mean, it, it feeds back into that, you know, the, the doom and gloom and, and I know we'll put this in the show notes, but you know, I, I found a research paper, you know, that, that actually demonstrates, you know, like the research that people have done on negative AI headlines, right? And there, there's something like, yeah, if you, if you analyze all AI headlines, like the majority just go negative.
And yeah, like that experiment I was telling you. Yeah. Yeah. And so, so, okay. So that's the first thing to keep in mind. Right. And then, and then, right. You have to just dig deeper. And I just had another thought of a new tool I could build, which is imagine if you just took the contents of the paper and put it in to chat GPT and said, come up with a headline, a realistic headline, you know, cause we've been comparing, but I'm like, well, what if, what if I had AI generate a new headline for me? Ignore this headline, you know, so I'd have to maybe like crop it out. So it has no way of like even seeping in there because yeah, this, this is obviously meant to, to catch attention, to provoke fear, to tell a story. Right. But, but yeah.
So the problem is if you just look at the headline and you send it off to somebody and say, see AI is terrible and they actually read it, they'll be like, Hey, I, I, thanks for sending this. This was actually really useful because it wasn't at all in any way to perform negative. So, I mean, well, let's dig in on some of the, some of these examples. Cause I think it's really important that people understand, okay, forget our perspectives. Like what are out in the world? What are successful companies doing to deploy? And what are they doing to deploy? And what are they doing to deploy? And what that really means is move pilots from a pilot into production. And the first thing to take away from 93% of AI pilots fail is we call them experiments again, cause I came from machine learning into AI and we kept that machine learning. Let's do an experiment first to prove that we can do this. And then from there, start talking about implementation and integrating it with workflows.
But that's what a pilot is supposed to be just is, does this work or not? And it doesn't work is really just as useful as to what knobs you go tweak to go get it to work. So it's a weird metric to just look at on its own to begin with. Um, but the first part is really one of the it's when you're even designing the pilot, when you're deciding what your, um, you know, what your experiment is going to be in the first place, what all, what everybody said here is, um, pilot design, design your pilots with the end in mind, start small and then aim big. And this kind of talks about us not trying to, YOLO against a language model with one big prompt. That's a little too big. Start off with the really small ones. When you think about using AI deliberately rather than opportunistically, like what helps clarify whether a pilot is actually worth doing in the first place in your perspective? So, I mean, I'm leaning back on, um, um, you know, this, another book that is coming to mind right now, you know, lean startup, right? So that's like a classic business, right? I've got it on my bookshelf. If I was in my desk right now, I could just go like this and point to it on the shelf. Yeah. So what's the primary lesson in that book is that learning is the most important, you know, asset. Like that's, that's the, that's, that's the goal is
to learn, right? So how do we learn if we're not actually doing right? So we, we have to get out there and do, and like the paper says that, you know, starting small, it's like, don't try and, you know, take on, you know, the whole business, just pick one area, right? And exactly what they say, by the way, uh, Kate, they, you know, they're saying as a matter of small as possible with the outcome in mind. So like, for example, they're saying, uh, be concrete about your outcomes. Uh, first of all, I hear a lot of people talk about like tests or, or they'll say POC and I know what they are actually doing. Like I would, I would say this, if you're running an experiment, which you should, you should run many experiments and most of them should likely fail. Otherwise you're not going to be able to learn anything. Uh, I mean, you learn if they don't, but if, if you're able to demo it to somebody, you might be able to learn something wrong. That might be too wide of a swath. There shouldn't really be a workflow. A lot of times these don't even need UIs. A lot of times the outcome of an experiment is just did this work?
Yes or no. And what did we learn? So for example, when you're, and then if you really want to tie that to ROI, it'd be successful. Here's examples of the paper says, for example, you might define a pilot by saying reduced tier one support handling time by 50% via an AI assisted triage or automate invoice processing to cut outsourcing, outsourcing, outsourcing costs by 5 million a year. That's they're, they're tying the goal, the specific thing they're trying to achieve to the ROI and why it's meaningful. I think if you just, I think that alone out of all these tips, you just define your pilots that way. And you're probably going to see more success because what happens if they're successful? Well, you can put them into production. I think what happens is a lot of people to find just fun experiments. And then even if they were positive and they worked, there was no connect the dots to how you actually put it into production. So there's a lot of people that are trying to do that. And I think that's where it fails right out the gate. Yeah. Okay. So two things are coming to mind. You know, first, there's another paper that, that I, that I looked at, um, from the Tata consultancy group, right? So they're a big, big, oh, I know Tata. Yeah. They're I've encountered them in one of our client deployments. Yes. Yes. Indian based, but global, you know, and so they did their own into the United States. Yes. So they did their own studies and they came up with a North American study, you know, an AI in business. And what they found was one of the key, you know, missing, right. It's like,
how do we do this successfully was missing was like, what are those metrics? Like, yeah, it's like, how do we measure? And so just that aligns perfectly with what that report is saying. No question. And by the way, different from evals, we're not talking about acceptance criteria. I mean, we kind of are, but you're really talking about if this works, how do we as an organization benefit? Yes. Right. Yeah. Not acceptability, but what our ROI would be in this effort. And the second thing is like, I don't know. So, so how familiar you are with the NIST AI risk management framework, right? Ha risk management framework. So NIST, so that's the National Institute of Standards and Technology, right? So this is a government organization and this, um, they came up with this framework. Now this was released in 2023. I think it's a voluntary framework, but it has four components to it. The four components are govern, map, measure, manage. Now this has to do with risk, but these components can be applied right on any level. And, and if you, they have this playbook, this is all free. It's all voluntary. And I think, uh, here's my future bet is that this is going to become more than voluntary.
So right now the state of Texas, we're in Texas. Yep. January 1st, a new law went into place. It's called TREGA, right? It's the Texas Responsible AI and Governance Act. We've got a couple of state and local government deployments here that we had to stop everything and, and see how TREGA. Right, right. So, so we're having to, so in TREGA, if you are adhering by the NIST AI RMF, you're kind of exempt. So all you have to do is follow this framework. Now, they built the framework in a way it's, it's flexible. Like, yeah, it's voluntary. You don't have to do anything. I know I'm kind of mixing things when I talk about laws and I don't want to get into too much of the weeds here. Laws and frameworks, but a law, a direct, like references a framework. Yeah. So it kind of fits perfectly. Exactly. But if you, if you really look at this and dig down, and this is what we do at Violet Beacon, right? We're helping people like actually adapt, like to some of these, you know, frameworks. So the NIST one, there's also an international ISO 42001, but, but what they do is they provide you with, it's really using, with the list of key metrics. So it's like, it gives you a whole guide. So you're not sitting there guessing in the dark, right? It's like, oh, you have the exact framework questions, you know, in a very structured way to help you, you know, um, um, make sure that you're doing all those things. But, but when I look at it, those four components about, you know, govern. So the four are again, govern. Govern, map. So mapping is like mapping the risks, right? It's like, well,
what, what is actually, you know, what could go wrong, right? So what, what do I need to pay attention to? Measure. No, measure is what we're talking about here. Like literally, how is this going to benefit? I mean, I don't know from a risk standpoint, but we could also, uh, map that so to speak to, well, how is this going to benefit your organization as well too? So mapping in both. And then manage is like going back and applying, right? So if you, if you measure and you come up with, oh, well actually we need to adjust, then you go back and you're making, it's like, so it's that iterative process. Yeah. And that, and that speaks to our eval framework guidance from before as well too. It's interesting. It also points to one of these other, um, uh, points the in, in this, uh, in the MIT, uh, uh, paper, they also talked about, uh, ROI modeling and measurement prove value or pivot fast. So when you think about a value from like a human perspective, uh, what signals tell you that an AI system is actually helping, not just being used, for example. Right. So, I mean, I think that does feed right back into that, into the evals, right. And, um, into the measurements, right. It's like, well, where have we actually set our goals? Like, wait, what are we actually hoping to achieve? And so, so that, that comes from like, yeah, having a, having a framework, like you should have a set of guidelines of what you're actually hoping to accomplish. Right. Oh, for sure. And, and I think that that does get
into that point where some companies are just like, oh, we got to implement AI to get ahead, to stay, you know, it's like, no, you need to be really deliberate about. Well, you can say, oh my God, we think our competitors are using AI. We need to do something, but then immediately go into, well, what is it about us that could benefit from AI? Yes. And that way, um, and actually what was really interesting about the, the measurement prove, you know, basically fail fast failing. Yeah. Like, is this working or not? If not, then let's, let's get out of here. Well, the easiest way to fail fast and be able to move on to the next thing is to pick really small things. Right. So we talked about that earlier, aim, you know, aim small. I actually heard this at a conference. I was at an AI conference. I said, um, you know, you keep hearing about AI is coming for your job or, you know, I think the thing that everybody pretended was their own quote, like two years ago was AI is not coming for your job. Somebody using AI is coming from their job, uh, attributed to like a million people. But this was an interesting take on it. AI is not coming for your job, just the parts of your job that you hate. And so to kind of back that up in this study, it actually talks about how back office automation delivered the clearest path to ROI. So, um, you know, uh, 1 million per year, they talk about in
terms of risk management outsourcing, like things like you're talking about, by the way, you can actually, if you're implementing risk management, you can save. But if you look at just like the tiny little things you can fix, like you don't have to replace a job, just replace the rote activities that can actually be, I mean, it's the same kind of thing we talked about back in the automation and the digital, uh, the digitization era as well, too. So, um, I mean, that completely feeds into what you said. And by the way, no better metric than the 1 million per year saved on risk management outsourcing. Hello. Um, so the other part too, uh, just kind of flying through a couple of these that are really important as well too, is, and I think this kind of speaks to the experience that you and I have about building custom applications and workflows. Uh, it's really about your, uh, integration, integration strategy, embed, don't isolate the AI solution. So AI that sits behind workflows rarely scales or on the side of workflows rarely scales.
Start with a contained workflow, kind of like what we talked about before, um, or a data source rather than integrating everything all at once. So hybrid architectures here. Um, so then when you look at it from a human intelligence standpoint, uh, what makes an AI tool feel like a natural part of someone's work instead of something that's bolted on? Do you think? Well, because that's really it. That's where the rubber meets the road on there. And this is where I think, you know, taking the responsible AI approach involves, involves kind of setting a culture and tone as well. I mean, we haven't like really touched on the people aspect of this, but those are real. It's like, yeah, am I going to lose my job? You know? And so with, again, with these frameworks, but also as a concept of responsible AI, you know, there's, there's mechanisms built into these frameworks in a way that you involve the stakeholders, right? So you need your subject matter experts, right? They're the ones who should be evaluating it, right?
They should be part of defining your pilots too, by the way, is what this paper is telling us. So that's important. So, so, so there's this, one of the concepts in responsible AI is around accountability, and that, and that comes from like, yeah, we need to be the ones making the choices about where we want to implement AI, how we want to let it make decisions, how we want to evaluate it. Like that all needs to come from us. Like, I don't think we need to, you know. Within your organizational structure too, as a matter of fact, Kate, one of the takeaways here is exactly what you just said. Organizational ownership. So define ownership. Someone must be accountable for the outcomes in production. And, and I think it could easily be done, like an IT job. Well, no, actually go to the individuals who, you know, experience the benefit or not, you know, in the, in their current workforce and let them actually take ownership of this. So, you know, real validation of exactly what you're talking about there. So I guess our pro tip there is when you're designing your pilots, assign it ownership of the pilot and accountability of its success to, or, you know, failures or whatever, to someone that can actually then do something about it and move a pilot into production. So again, mapping to some, you know, the right people there. And leadership plays a big role in this too, because they're the ones setting the tone for the culture. And that's a big part of both of the ISO
40, 2001 and the NIST is like making sure that leadership is, is involved, right? They are ownership in this as well, because it's going to come down from the organization, from the lead. Yeah. The paper actually says successful organizations decentralize authority while keeping accountability, i.e. a line manager ownership plus executive sponsorship involved. And by the way, just to toot our own horn, I, I couldn't get through a whole podcast without quoting this one. It indicates that strategic partnerships achieved 67% deployment success while internal only builds only 33%. And employee usage rates were nearly double in externally developed systems.
We were, we're here at a podcast studio where they just told us that they're, you know, heavily involved in tech and they started by building it themselves internally. But then when they moved out to partners, they were able to get it into production. So make sure to check that out. I think that's a really, you know, a similar type of approach there. Also, I think you'll appreciate this. Those end users who we know are so important when you're building your workflows and your custom applications out, the paper identifies them as prosumers. These are employees that already are using ChatGPT or Claude personally, and are, could be key champions on a bottom up sourcing of the use of internal. So one of the pro tips I'll give right out of the gate, and I tell, I say this all the time for executives that want to know how to get into this. Number one, take inventory and who, what prosumers you might already have within your organization. They might surprise you. It might be somebody that's completely low tech in their work efforts, but they're your, your biggest pro user of AI. And you can actually learn from them and make and get them involved in some of these pilots, right?
A couple other thoughts to kind of get through some of the big ones here to make sure we're given some really good nuggets. This one, I know you'll appreciate feedback loops. As an, as an ML guy, the world of AI, really what fascinated me was human reinforcement and having humans in the loop in the most effective manner. And I know this is a critical part of your human intelligence positioning. So in the paper, it says make AI systems improve over time instead of stagnating. I don't think most people know you can even do that. Right. By introducing state and memory into your AI deployments. So for you, when you look at like human intelligence and humans and AI working together, what kind of feedback mechanisms do you feel are most natural for people to be able to give feedback and probably more, most useful for the systems to learn too? Right. I mean, we're used to thumbs up, thumbs down, but it doesn't stop there, right? Yeah. I mean, I think it, it of course has to be built into the workflow, right? So it has to be natural, right? The way we think about giving feedback. So, you know, even if it's just, and that's the great thing about with LLMs, right?
It can just be generic notes. It can just be thoughts like, how would I give feedback? You know, but having that, that, you know, binary, like, yes, no, actually works really well for evals because there should be a very clear definition. Yes, this is right. No, this is wrong. Right. And so, so giving that kind of feedback to an LLM, like you talked about, like, yeah, in training, right. Is, is actually extremely valuable, but I think from the human perspective, always making sure that there's a place to give feedback. Right. And so even if that's, you know, a mechanism in organizations for, for, for, you know, um, you know, a place to log in, you know, like, you know, a place to log incidents, if there's something that happens, right. And you're like, oh, wait, this wasn't supposed to happen. Like all of that.
Super. And trace that's going to feed back into your traceability. Cause I can tell you that my development team, when we built traceability, like we literally have, first of all, this is kind of interesting. We used to have thumbs up, thumbs down, and we completely got rid of thumbs up because they should all be thumbs up. Right. So what we really want to know is when the workflow or the model responses diverge from your expectations. So we went with just thumbs down. Right. That, uh, you put your notes in and in our trace logs, we're assigning it exactly to the turn in the conversation. If it's a chat where that happened and it really gives our developers everything they need. And it gives our users the way to actually, uh, explain what they were expecting and maybe, you know, what the gap was there. Um, and, and, and some of the stats that support this Kate are from this paper, 95% of enterprise AI pilots fail. Oh, I'm sorry. That's the headline. I'm just going to stop right there. Uh, but it says, because this is why you should read papers because systems don't retain feedback, adapt to context or improve over time. So we're here to report. They absolutely can.
Right. And it's a fundamental mechanism to building a good AI systems. They found that this paper found that two thirds of executives now required AI tools that learn and remember feedback. Isn't nice to have it's the line between the 95% of these pilots that fail and the 5% that don't. I mean, that is, that's really it. Right. And I guess people don't know that you can bring feedback in. I'll give a really practical example. If I may, um, a software team that I'm working with, who's building an AI tool, um, would only approach a turn as in a user asking something, and then they just go find the answer. So honestly, it wasn't really like feeding into the nuance of, of what the AI nuance and what LMS can really bring into a workflow instead. What it was, it was, it wasn't really like feeding into the nuance of, of what the AI nuance and what LMS can really bring into a workflow instead. What it was doing is looking at it as a question answer machine. And by the way, I would caution if you think AI is a question answer machine for business in particular, you're doing it wrong. I'm just going to take a hard line on that. So in this case, they said, well, the user says, well, what's my best performing product, for example, and the AI is giving them an answer. And they're like, well, that's not, I don't, don't consider those to be my best products. And then, so the software team was like, well, why don't we have a little window that pops up where they can type in what they mean by
best. And I said, hold on, I'm just going to type in what they mean by best. And I said, hold on, why not just ask, have the LLM be trained, you know, uh, pro instructed to ask the user when there's ambiguity. So they introduce a follow-up question into the turn, which is standard. And they saw their eval metrics. Actually, I think in this case, double, I just heard them a couple of days ago. So by simply allowing the LLM to follow up when there's ambiguity and ask the right question, I mean, language models have a bunch of right answers, right? I mean, yeah, we kind of have to tune them properly. Exactly. Exactly. We're just, we're learning how to use this tool, right? And it's learning, it needs to learn what we want. So the more it can learn about our users and that's what this whole feedback loop thing is about, the better it's going to perform. I mean, I can tell you from the field, that's definitely the case. So, so far what we've been telling decision makers and executives is design your pilot with the end in mind, integrate and don't have it isolated, like not a chat on the side or have it integrated into your workflows. By the way, we're going to talk about that in a little bit, but I think it's going to be interesting to see what's going to happen in the future. And I think it's going to be interesting to see if we're going to talk about that in the future. And I think it's going to be interesting to see if we're going to talk about that in the future. And I think it's going to be interesting to see if we're going to talk about that in the future. And I think it's going to be interesting to see if we're going to talk about that in the future. And I think it's going to be interesting to see if we're going to talk about that in the future. If you go into your systems of record and do integration, you'll likely see more success. That's kind of a, Oh yeah, that kind of makes sense thing. The higher
use applications will get higher use, you know, outputs. But I thought that was kind of an interesting thing to put together, but they said for high stakes workflows, a deterministic spine with the AI at the edges outperforms pure generative autonomy. That I think is a huge takeaway here, right? We talked about feedback loops and that leaves us with that one and then organizational ownership as well. But there's a couple that there's one that I want to throw in as a bonus. Um, because we're data people. And then I know you had another bonus as well too. We can, we can end with these two bonuses and a little bit of a wrap up here, but here's what my bonus use cases that I found came. I found another MIT. I was literally sitting at, you know, entertaining myself at dinner by myself last night. And, uh, I found a report, some headline, I don't know what the headline was. I should have grabbed it, but it got my attention. It was about an MIT report. And this report said around 60% of, um, it says most companies are spending aggressively on AI, but the majority are not seeing meaningful financial returns. Of course that caught my eye around 60% report, minimal revenue growth and cost reduction, despite heavy investment. However, this report noted that a small group of AI leaders stood out roughly five times the revenue gains and three
times the cost savings compared in this report to the ones that failed. What did they have in common? Well, in their case, their key differentiator Kate was foundational readiness. The future bill companies have to invest early in data infrastructure and core capabilities that allow AI to work responsibly and reliably. So what we were saying there is they're going back to basics. So the CEO takeaway here, the decision maker takeaway I would offer, what are your thoughts on this is go look at your data in some of our early deployments. I admit I mistakenly thought LLMs could overcome bad data. Not true. So now we go, go in and do a data discovery, take inventory of the data. Is it clean?
Is it reliable? Can we address these things? And sometimes the answer might be, we need a multi-month project just to deal with our data and our capture before we can actually rely on LLMs, because it's kind of a garbage in garbage out thing, right? I mean, this is no different than the custom business applications we've created in the past, right? Yeah, yeah. And you're absolutely right. You know, I mean, in another report, the same one I mentioned earlier, the Tata consultancy report, that they were quoting the same thing that, you know, in, in looking at where companies are in their, in their progress of, you know, adopting AI, that there was a significant portion in just preparing their data, right? Because, and so, so back to that MIT report, because I think they only looked at six months, right? And it's like, that's also a short period of time.
And so that's gonna, you know, there's, there's all these little factors, you know, there's like this term interrogating research, you know, and like going in and looking at all that. Another thing about the MIT report that I discovered when I was digging a little deeper is it wasn't actually produced by MIT, right? There's a company, not a company, an organization called NANDA. Yeah, it was a NANDA report. Right. Yeah. Yeah. Is that technically part of MIT? No. It's, it's formed with people, but what NANDA is doing is they're working on building standards, right? Similar to MCP, if you heard of MCP, I mean, of course, MCP, anthropic, A2A, right? So these are the standards that, that again, like, this is when I say we're not in the, the...
Assembly line. Assembly line, yeah, right, like, the standards are still coming, coming together, so we don't, we don't have the perfect ways to implement these tools, and that's what a lot of these, you know, people are developing. So, so I thought that was kind of interesting back to that MIT report is thinking, like, okay, well, what was the perspective of the people, you know, writing this report, you know, obviously coming up with, like, here's how, here's, here's the direction we need to be headed, and we're not there yet. Well, in fairness, they're, they're, the way the headlines were picked up on their own paper, I mean, they did say Gen AI divide, so they were trying to say, we're creating the need for standards and an approach, and a lot of these speak to the types of standards that they're looking for, too.
So another thing we'll throw in the show notes there. Yeah. Was the Tata article, the one that you were telling me this morning? Yeah. Great. I'm glad you got a chance to mention that one, because I think that one was chock full of reinforcement here. Yeah. Kate, how do we bring this all home for folks? Let me just do a quick, like, summary of what we learned here. So the decision makers out there, this is what we're saying. Well, first of all, don't just read the headlines, read beyond that. Right. But some of the rules and the takeaways that we see in the successful deployments were, well, working backwards, start with your data health first, spend as much time as you need.
Getting your data healthy and structured before you bolt gen AI on from there. That's a big one. Also, you cannot express the thesis. If you cannot express the thesis as a CFO-style equation, meaning what's your literal financial gain and output to this, it's probably not a pilot yet. You probably haven't formed it yet. Next, for high-stakes workflows, a deterministic spine with AI at the edges will outperform pure generative autonomy. I think a really important approach. And then finally, well, actually, if you're not embedding AI into the system of record adoption will be cosmetic. And then finally, build feedback capture into the UI, not into a monthly survey.
Meaning, hey, guys, how effective was that AI that you used? Do it at the moment, like you said so wisely. Right there at that moment where the feedback actually emerges. And then I would actually add, find your prosumers. Go in on Monday morning and say, hey. Hey, everybody. Raise your hand if you use ChatGPT. I want to have a conversation with you about it, right? So in closing, really, this conversation was intended to not be about AI optimist or pessimist. It's about being operationally serious about success with AI. And it would be irresponsible for us to just talk about headlines and people's approaches to them and their interpretation without getting into the meat of it and looking constructively at how people can be successful.
So one of the ways they can be successful. Right. Right. Right. Is to be able to work with outside. We saw, hey, twice the success rate, two-thirds success when you go with outside sources. So if people want to find you and engage you at Violet Beacon and really, because a lot of what you're offering from a consulting standpoint are starting points pre-pilot. So talk a little bit about how people can find you and how to get started with you on those things. OK. And I wanted to close. I have one more thought, because I realize as you were summarizing everything, we've touched on almost all of the pillars. Yeah. Yeah. The pillars of responsible AI. Right. It's accountability.
It's transparency. It's, you know, reliability. Right. That's high performance. Those are the evals. But some of the other, you know, privacy, security. Oh, of course. We didn't touch on those. And. But. But. Which, by the way, I think is what people think all, that's all responsible AI is. Right. When it's not. It's the other things that you mentioned. Right. Right. There's all these other things. And I think what draws me so much to responsible AI and to these frameworks. To the NIST framework and the ISO 42001, which again, this is what I'm using to help clients because it's a grounding. It's like, here, this is a process. This is a framework built by governments, non-government organizations, private individuals, private companies in a way that's like, okay, there's some basis.
There's actual grounding in this, is it gives you something to fall back on. It gives you those boundaries. It gives you those breaks. Right. And so that, you know, like you have a safe boundary because the fears are real. And what I love about the definition of AI in the NIST. Is they use, there's a long definition, it ends with the goal is to create trustworthy AI. Absolutely. And that's what responsible AI is about. It's not ethical AI, which is a whole different thing. Sure. Holder and show. Oh yeah. But responsible AI is about creating trustworthy AI. And these, and I think that's what we've been trying to get to is like, here are the tools, like there is a way to do it.
The headlines are missing the point here. Right. Recall. So I wanted to throw that. It's all about trust. As a matter of fact, our headlines are fear. We covered fear. Right. So how do you gain trust and how those become the playbook for successful deployments? We did it. Right. But I will say my trust equation, by the way, is relevance, accuracy, and evidence. Something we can talk about in a later conversation, but trust isn't just one trust. How do we get a trust? You have to make sure you have accuracy, but it goes beyond that. Is it relevant to me and contextually relevant to me? And then can I see within the UI evidence of where this information is coming from?
So to kind of bookend. You can bookend the point that you're making. And so that seems to be a good starting point for people strategically. And if they want to start with you, where can they find more information about how to get engaged? They can visit me at violetbeacon.com. I'm active on LinkedIn. I'm working on activity in other places, but those are the best places right now. And I do have a blog called Human Intelligence. I love it, by the way. I wanted to make sure you got that out there. Check out Kate's Human Intelligence series, really. Yep. And you share that on LinkedIn as well, too. Yeah. I appreciate it a lot. Really a lot of the conversations you and I have, I see that in there as well, too.
You just dropped on this morning, as a matter of fact, an exploration on the headline reading is part two. So great follow. And of course, I'm Chris Hippolyte. You can find me @isolutionsai pretty much anywhere. This is really fun for me, Kate. I've been looking forward to having a conversation with you. It's fun to be here in Austin, your native area. I look forward to our next chat, certainly, but I really appreciate you having this conversation with me. We could talk for hours on this. I know. For sure. This is awesome. Thanks for having me on, Chris. I'll see you next time. Absolutely. Cheers.