Richard G. Greenhill, DHA, FACHE (Host): Welcome to the Healthcare Executive Podcast from the American College of Healthcare Executives, providing you with insightful commentary and developments in the world of healthcare leadership. To learn more, visit ache.org. My name is Rich Greenhill and I am today's guest host for the show.

So, very excited, about the topic today. And, we're going to speak with a national leader on implementation of artificial intelligence and some of the key aspects that leaders should be aware of if they want to be successful, and have sustainability. It's a pleasure to welcome, Healthcare Technology and AI Expert Mr. Edward O'Connor. And Edward has spent more than 20 years as a thought leader executive and consultant across basically all sectors of healthcare. I met Ed, many years ago, as a consultant on the same team. And so he's worked in public, private, and Veterans Health Administration, as well as military health.

So welcome to the show, Ed. It's great to have you.

Edward O'Connor: Good to see you again, Rich.

Host: Yeah. And so, to the audience, we certainly hope that this will be insightful for you today. Ed has a tremendous amount of experience. So, Ed, you've been doing a lot around AI and we've known each other for years. So, what do you have going on these days related to AI and IT and analytics?

Edward O'Connor: Well, step one, I can't believe you called me a thought leader. Don't ever call me that again. Not sure what that means, but it's probably not me. I am lucky, as you know, to work in a lot of different sectors. Most of my day job has been supporting defense health agency, military health services, which is relatively new to me in the last two, three years.

 But I also get to help with some other programs that are on the civilian side, get to help with some programs in the private sector. I'm the recipient of a lot of Phone Your friend requests, which is enjoyable for me. I think just like when we worked together, a lot of times I'm the most technical person in a room of executives.

I'm often the least technical person I think in a group of technologists. So I'm able to bridge the gap pretty well, which I enjoy.

Host: That's great. Well, I'm aware of what you're doing and certainly you're making a difference. So we'll get right to the topic today. So, as someone who's worked in implementation for many years of different things, AI implementation is just like others, right?

Whether you're implementing a new EHR or you're working on a performance improvement project. And so, as we look at AI, a lot of organizations are thinking in terms of pilots, they pilot in a safe environment and then bring it over to connect to the legacy system and, bring it up to operations.

And so there is a thing in regular change management, right? Quality improvement called pilot fatigue. And so, I know that that's probably an issue, operationalizing AI. And so what do you think about pilot fatigue as far as AI is concerned? Is it similar to what we've experienced in other areas? How do you think it impacts the journey from proof of concept, let's say to implementation of whatever the tool or method is?

Edward O'Connor: Yeah, it's definitely a thing. I would say it is both the same as it's always been, but I would probably say it's as worse as I've ever seen it. So as you go through, I'd say every decade, there's some technology that really gets pushed hard, and generally ends with a bubble or a fair amount of disappointment, whether that was data warehousing or that was EMR deployments or that was blockchain was briefly like the source of much disappointment and random pilots. AI, the pressure is as high as I've ever seen it. So I am seeing everyone from board members to executives, to technology leaders, just no matter what we're discussing, the response is just, okay, how do we apply some machine learning to this, some AI to this, some automation to this.

So the pressure is just endless. And the result of that is every project is getting pressured to do some AI, often by folks that aren't sure what they mean by that. And sometimes, I'm not sure what I mean by that in the sense that the definitions change fairly regularly. We've decided to use AI as this big umbrella term. But people just want to see ambitious pilots. I would say two years ago, I just thought that was great. Like there was this willingness to just give this a try. I felt like the learning curve was so steep. We were all just going to have to try this stuff together and see what works, see what didn't work.

 What's concerning me is that we're exiting the pilot stage successfully. So it's been a couple years. By this point, the bubble should have either burst or we should have figured out okay, these are the use cases that are pretty easy to get off the ground. These are the other use cases. They're just not going to get off the ground.

They're bad fits. We should also see like just a steady percentage of things coming through a funnel where, okay, we tried 10 things, two of them worked, and we're starting to see a pattern on which things are working, which things aren't. I don't feel like we're there yet.

Host: It's very interesting, as a data scientist, I think about, so this is interesting there, there's so much to know about AI, right? And, I tell people all the time when I'm speaking at conferences and such that it's not like you go pick the milk you want out of the store, the brand and you use it, right?

There's pros and cons to everything and I, I think to your point, it does seem like just an endless trial and error period already in a, situation where the workforce is stressed, margins are thin. Funding's getting cut. So it feels like we're headed toward an inflection point, of a little bit of a, a disaster with the workforce, tiring them out.

Just from, let's try this today, or, let's try that, or, or, we're going to do this and then we're going to train you on this. And, AI even for someone who understands it deeply, it changes a lot. I mean, we were doing traditional AI back, a couple years ago, just machine learning. And now we've got the general AI, which is an expansion of that and all types of learning inside of that. Right? So, like myself, you've worked across all sectors of healthcare. So, you I've been in defense, I've been in VA, I've been in military and private and public, whatever, is a use case a use case? So if, I was looking at department defense or VA and thinking about a for-profit operating system or the safety net, as an example, are those things synonymous? And can what works in one sector be easily transferable to another sector in your thought?

Edward O'Connor: So let's split that into two questions. Both are really good questions. So the first question is, I would even take a step back and say, what's a use case? And the different groups, not just sectors, but just on the project level, is their operational definition of a use case similar to the project, maybe in the same department?

The quick answer is just no. I was on a project just yesterday where I just had to sit the team down, and explain to them, they were saying use case and they're doing, use casey stuff in three different functions on just one relatively small program. Maybe medium-sized program.

So you had UX people, usability people, user experience people. They're doing journey mapping. They're doing personas, they're doing human-centered design. From their perspective, they're laying out use cases. That's not a use case. What is it? You have the software development team. They're living in this agile world. They're thinking in terms of user stories. A user needs to be able to do this for this reason.

Because they need to chop their work up to get something done every day, every week. You've got technology architects, which are the people I'm mostly trained as. They're off kind of in a silo. They're doing formal use cases, using formal notation and universal modeling language, like a use case has a very specific formal definition like capital U, capital C.

 And then there's a business analyst somewhere else actually talking to end users. Just trying to sketch out, block diagrams or workflow diagrams that make sense to everybody. What struck me is on this team is maybe 25 people total, they still managed to create four siloed groups of people doing, call it requirements, definition, or, you requirements visualization.

They were doing them four different ways. None of them were too concerned about who looks at their product, so they were very concerned about creating their product. They weren't really looking at, is anybody using this? Are we looking at it every day? And so, you just had to convince management, like, I need to take one person out of each of these four silos and I need to put them on the same team. We need to really just make a decision as to what's our deliverables, what are our artifacts going to be, and we need to be merciless about it. And that's especially true as we're using AI on these actual teams to write code. So the importance of journey mapping and requirements, like if those are really good, these AI code generators generate relatively good code.

 If that stuff is all over the map or there's four different versions of it, I mean, we're just hosed. The code is terrible. It's just creating more technical debt. So that's sort of the first question is just people are now figuring out, Hey, what do we want our definition of a use case to be? Especially now that a use case is becoming a prompt for AI generated code and system development and system architecture.

And then on the other side, like from a business standpoint, also use the term there to say, okay, you've got a business problem. Is AI good for this? And what kind of AI is good for this? And again, it is just all over the map. So the quick answer is no, it's not transferrable at all. I think anytime I hear a use case is a use case, this is usually a vendor who's trying to sell us the same tool or the same model to multiple sectors. They're trying to expand from one sector to another, which I identify with. You gotta sell stuff, but it's closer to saying a scalpel works. It works the same in surgery and woodworking. Like it's a cutting tool.

I can make a table or I can conduct surgery. The same use case, same tool. Like, it's just not, it's so far from the truth and it's, it's getting further from the truth. Prior to AI and all these advanced tools and speed of delivery, use cases were more universal, technology was more universal. It was still a lie to say, it's going to span all the sectors, but it wasn't as big a lie as it's become.

Host: That's interesting. Particularly the end user. The result, right, of what you're creating from the use case, in that we have to think about how end users are going to react or use it. And so, as a change practitioner, I often had the difficult task of managing, leading organizational change and dealing with the turbulence around implementing initiatives with my leadership teams, and as much as we would like to think artificial intelligence is different or the tools are different, it really is another change leadership issue. And with this, I believe that there is, like in other things like Six Sigma and the other things that I've, I've doing, there's a high potential for backlash.

It just really is. And so why or how do you think tools like Ambient Documentation, for example or AI scheduling, how can that, as an example, maybe trigger some unintended consequences in culture and, disrupt the culture that's trying to do good work?

Edward O'Connor: Well, I mean to use those two examples as specifics, they are triggering backlash. It's not a risk anymore. It's materialized, it's an issue. People are getting broken down by these tools in real life and they're rarely succeeding. They are in some cases, and the answer is very foundational. These tools touch on the identity of what this person does for their job. It touches on their autonomy. It hits the very basics of trust.

These tools are directly, taken a shot at people's identities, day to day. So, you when AI starts, Ambient Documentation, starts documenting what clinicians say or reshuffling their schedule; it crosses a line from support into surveillance unless you're very careful about how you implement it. So, if clinicians think AI is making inferences without their input; they'll aggressively resist. And I would say they should resist. It is their job to resist. Real world example, I was advising on a project for an Ambient Documentation trial.

I've actually advised on a few now in rapid succession. And the one thing they had in common is the clinicians were already pretty well burned out. Like these people have, post-traumatic stress from EMR deliveries, from meaningful use. Like they just feel like they've been taking a 10 year long beating. The technology worked pretty well. Pretty accurately transcribed, structured the visit. But in all these cases, it over summarized in pretty subtle ways. They would omit a nuance around a patient effect or how uncertain the patient was or the conversation was, and it was also capturing things that the clinician might say, or the patient might say offhand that was ending up.

 Clinicians are very concerned this stuff was going to end up in the legal record, end up in the chart, and create a downstream impact. So trust was really the issue. The difference between the successful pilot was we really started out with clinicians in the successful one and the clinical operators, the nurses, and got them bought into it.

And we really focused on integration to turn the tools into what I would call an invisible intern. It wasn't a big brother that's acting smarter than them. It's an invisible, dumb intern that's taking a whack at the notes to save them time. And then implementing that in the EMR in a really low friction way.

So the notes were just going in as the clinician was typing. It was very clear where the Ambient Documentation app was sure of what was being said and where it thought it had a nuance and it separated that. And then contrast that with the failed pilots. These were where technology people, we got in a room, we came up with pilots.

Speaking of fatigue, we tried to get them pretty well baked, and we did this for good reasons. We did it because we didn't want to create another burden on the clinical team where they had to get in the weeds with us from day one. But the result was that as we hand them something 80% done, we left the 20% of hard stuff for them.

They didn't trust that we did the 80% right and they shouldn't because the team really didn't. And so I think the real lesson here is you've just gotta take a step back and say, do we actually have time to engage people from day one? And if we don't, maybe this isn't a good idea right now. Or maybe we need to figure out how do we free up that time for real, not just add it to people's days.

Host: That's a good insight, Ed, and good example. I just gave a talk in Ireland, and we talked about implementing artificial intelligence and, risk, legal risk, patient safety, and those types of things. When we think about what you just said about the cultural piece, surveillance versus support enablement, there's always the notion that leaders set the tone for how AI is going to be used, right?

So if we're just thinking about revenue expansion, revenue generation, which is important, we gotta keep the doors open, right? That's one thing. But, but how do we ensure that, we fail fast and learn quick? And that's a challenge in itself, right? Fail fast, not damage the culture, and in the risk management space, whether it's financial, legal or, otherwise, there's a concern of who's accountable when AI tools fail.

And, that's something I do a lot of speaking about from a quality standpoint in particular because patient gets harmed. We're going to do a root cause analysis or whatever our process is to figure out what happened. So why do you think that this is also a cultural risk, kind of aligned to what you just said, with unfortunately, who do we blame?

We don't like to say the word blame, but that's how the process is still set up to evaluate what happened and assign where there's accountability. So.

Edward O'Connor: Yeah, I mean, we are in a business that blame is important. Accountability is important. That's how it's structured and how it should be structured. And I think the accountability, call it an accountability gap, it's fundamentally a governance problem. But I do think there's also, underneath the governance problem is a misunderstanding of just how the technology works.

So, you've got a model trained on historical data. You're deploying it in a live clinical environment. It's almost always now, and this is different than a couple years ago, where I thought we would have more explainable and easier to interpret models, but the black box models have just advanced so fast.

This idea of, well, we don't have to use that. No, you, you kind of have to use them. They're that much better. So we've created this explainability challenge. And when something goes wrong, whether it's a wrong suggestion or a missed flag, or there's a bias pattern, it's immediate finger pointing in terms of invariably the clinical team's going to say, Hey, I told you this is hard.

There's nuance to it. You can't just make these difficult decisions. More goes into it than what you see. More goes into it than the data. And then you have the technical vendors, which sometimes may or the data governance expert, invariably we're going to blame the data. Because nine times outta 10, that is the underlying issue.

 But one of the real issues are the delays in the quality improvement process. In terms of like one system I was working on, we had a pretty simple model. This is kind of a low stakes pilot where it's embedded in a patient referral system. So it's really just looking at triaging and flagging patients where, hey, if, we've got an unlimited number of people we could do outreach to, how do we put that group into an ordered list, in a way that is cognizant of their actual risks? That's hard. One small mistake happened in the data format where EMR was still updating some ICD9 to hybrid ICD10 variance, which is still going on all over the place.

We didn't notice the problem until weeks later when we saw just some adverse outcomes coming in. So, vendor blamed the data, clinicians blamed the model. Leadership team realized they're really not plugged into this level of problem.

 And they're really not capable of adjudicating this. Like they weren't capable of stepping in the middle and ending the finger pointing. Was a key issue. In the end, that's the executive's job, is when you've got two sides pointing at each other. You've gotta step in the middle. And yeah, you need to figure out who's more wrong than the other.

But mostly you need to get those two sides to where they're not operating as sides anymore. That's really hard if no one truly understands the technology in the middle. And very few people do. As you know, I'm in the technology circles. I'm considered an adequate engineer and adequate programmer, real technical people make fun of me.

But I spent a lot of time on interpretability of models and explaining models and getting people to trust models. And it's hard to even get me to trust this stuff. Because, even the people that write this code, if you really dig in and say, why do you think it came out with that answer? They're like, well, here are the different ways we can explain it.

So that's not really what I asked you. Like, where do you, where do you really think it came from? Not asking you how should we create a surrogate model to explain the more complicated model? That's a technique. I'm saying, does it make sense to you that it made that recommendation? And the answer is often no.

They're just like, we don't completely understand how some of this stuff works. It's beyond anybody. And a lot of people are still surprised that it works at all, especially with LLMs. Even the people programming the LLMs, the Groundbreakers are still a little startled that it worked.

Host: I would agree with you. You know, I've done a rung less of coding than you, and I don't trust always the things that I code. I code them aside sometimes for fun and you don't know where it came from and, you're trying to go back through your own code or, or especially if you ever use the large language models to create code for you and you, you know, or put into the model to help you. So it, it is a challenge for even people that are probably well, well established. To your point, I think that the trust comes with model validation is something I've been talking about for a, a couple of years now, that's really the key.

And so you are the model validator. I believe that that function needs to be built into every organization in how they use the models and so taking them off the shelf and using them without having a model validation function, I believe is very risky. And it's starting to show up kind of in your example.

So, well, Ed, this has been a great conversation and we're getting down to the wire here to close out. So, as always, you're a good friend, but I value your insights as well, and I think that the, audience will as well. So I wanted to see if you could give us a couple of your nuggets, here around just digital transformation and artificial intelligence and, thinking about how to operationalize AI in as far as into legacy systems, and you how to train and bring a really, what I would call a tired, skeptical workforce through the transition to be at a place where they can constantly change. Because I will tell you, I'm worn out from the darn stuff and I, I look at it all the time and I love data science and AI, but, it is draining to me.

I can't imagine being a clinician or a leader having to constantly keep up with the change that happened two days ago and then two days later, and then another variant. So what are your thoughts there?

Edward O'Connor: Well, a lot of these lessons, these are the same, these are universal. We've had these lessons since the term Health IT was first coined and I feel like we're still not listening hard enough. So let's just step back, seven or eight years. Okay, so the leaders have their EMR, they're putting the data in the system. At this point, they're spending less time with patients. They haven't seen the value from it. And then we try to implement these large scale, kind of regional public health or population health initiatives. And the lesson we didn't learn quickly there was we spun up a whole different set of systems and then the team had their EMR and they had a separate population health system, and a separate case manager system.

So we turned all these folks into case managers that needed five screens. And we knew the right thing to do was to just start embedding the most important functionality into a single user interface. Like they're not going to go to this whole pop health separate system. It's not going to happen. But if you could just take the three most important data points, you're trying to get outta that system and into the clinician, put them into the UI of the EMR.

We knew that was the right thing to do, but it was really hard to integrate with the EMR. So we didn't do it anyway. We just spun up more systems. We're doing the same thing with AI. People need to acknowledge all healthcare systems are legacy systems. They're all clunky. We're all learning this together.

It's not that mature of a sector in terms of technology or user experience, but what I would encourage is, try to replace or re-platform the whole stack. Don't try to blow up your infrastructure. Just figure out what you can embed directly into the existing user experience. No new logins, no new interfaces.

Just again, to go back to an earlier statement, just kind of a dumb intern that sits in the background and offers information right into your existing workflow. But for every pilot, every project, really sit down and look for low friction, minimal friction approaches. Which is again, a lesson. I don't know how we're, haven't learned that lesson. Because we've implemented new systems, we've watched them succeed or fail.

They've only succeeded when we implemented them, where it was led by the clinical team and it was low friction, user interface. Everything else, people are just filled with remorse on. They lost value.

We lost patient safety, we spent more money. It's kind of the opposite of the triple aim, kind of the triple fail. It just didn't work. And I feel like we are making that mistake again. We're just saying, oh, well, we'll just replace all our data and create a new data lake. And people have to go into a separate AI based system. It'll end up replacing everything, but what'll actually happen is it'll just be yet another system and that'll be a shame.

Host: Well, this has been great speaking to you, Ed, and great food for thought for our, listeners here. And so, this has been a Healthcare Executive podcast from ACHE. For more information, visit healthcareexecutive.org and thank you for listening. Appreciate you being on Ed.

Edward O'Connor: Thank you.