FinTalks: Creating Legal Documentary Efficiency with Artificial Intelligence

Do you want to know how AI can solve real-world challenges and drive business growth? This interview takes you behind the scenes, revealing the workflow between developers, legal partners, and the tech team. Uncover the challenges faced in prompt engineering and verifying AI output. The discussion also delves into the experimental nature of AI development in Python, contrasting it with traditional application development. Explore the unique balance between the human element and AI within their application and gain valuable perspectives on the true worth of investing in AI.

In this episode, we sit down with Jonathan Poulter, Chief Product & Technology Officer at Verifi and Anna Samborowska, Business Partner for Financial Services Practice at Future Processing, to explore the fascinating journey of integrating AI and its impact on modern business. Enjoy the discussion!

Anna Samborowska: Hello everyone, welcome to the next IT Insights discussion. Today we are going to dive into something that has been buzzing every boardroom for the past years. Today our guest is Jonathan Palter, Chief Product and Technology Officer at Verify. 

Jonathan Palter: Great to be here. Thanks for having me. 

Anna Samborowska: John, artificial intelligence is not a buzzword anymore. It is a real game-changer as it helps every new business flourish and it helps to solve actual business challenges. Could you please share how do you use it and utilize it in your Verify products? 

Jonathan Palter: Yeah, sure. So, we’ve identified a real problem in all sorts of industries in the verification of documents. So, um, we’re really concentrating at the moment in one particular domain, this equity capital market. So, think about the documents you need to verify in that industry, so IPOs and mergers and acquisitions. So these are transactional documents which contain hundreds if not thousands of factual statements, and it takes a team of legal experts to pour over those claims and then look at private repositories or over the public web to find statements which corroborate those claims. And this is a really inefficient, time-consuming process, and this can lead to bottlenecks, it can lead to cost overruns, it can also just… just provide general time pressures, right? 

So we’re using technology, and specifically artificial intelligence, to actually cons… to really bring down that time component and give that time back to these legal, uh, domain experts so they can focus on real high-value activities, right? So they can do things like, um, strategic thinking or client advocacy. 

So when we were thinking about building the Verify product, one of the main, strong core design principles that we had was: how could we embed this artificial intelligence deep within the product so when it percolated up and was exposed to the end user, it was highly intuitive and it was very much focused to the domain at hand? So we have to think long and hard about how do we embed the artificial intelligence deep within the product so it runs in the background seamlessly without disrupting the end user’s workflow. Because we see artificial intelligence as an activity, at the end of the day, which augments the user’s workflow. It doesn’t, uh, doesn’t replace it—and not for the foreseeable anyway. 

In order to do that in an optimal way, you really got to really appreciate the workflows that are being conducted within a specific industry, uh, in this case, uh, equity capital markets. And this is what we’ve been doing with, um, Future Processing for the last couple of years. 

Anna Samborowska: And from the delivery perspective, you know, focusing on the behind-the-scenes when it comes to your work and the way your product work… there are developers, there are legal partners, and there is rest of the team. Are you facing, or do you face, any everyday challenges in regard to AI, like creating prompts or, more importantly, verifying them? 

Jonathan Palter: Yeah, it’s an important point. And with any AI initiative, you have to start with the data, okay? So the data is the fuel for the models, and therefore it’s really important to have domain experts that really understand the problem domain. And so they have to be really, really comfortable with the data sources that we interact with, the documents inside those data sources, the… the type and shape of that data within those documents, and how they all relate to one another. So you need to have that really strong data foundation, and they also—the domain experts—have to work really closely with the AI development team. 

So in order to do that in a… in a really efficient way, it’s so important to have a fairly mature LLM ops capability. And what I mean by that is the ability to develop, deploy, and run these large language model applications through structured, scalable, automated operations. Because what you really want to be able to do is to run lots and lots of experiments, and you want to be running this really tight feedback loop of build, measure, learn. So you’ll be running an experiment and you’ll want to understand whether that’s improved or degraded the artificial intelligence, recalibrate, and run again. So there’s literally thousands of these experiments that you need… need to run, and you have to do it in a really efficient way. 

And also, in order to be able to do that, you have to have a good sense of what does it mean to be good? What does “good” look like? So effectively, these are your KPIs, right? Or in a more data science parlance, it would be called objective function. And I think it’s generally well understood with verification what that means. It’s two things really: one is, have we found any statement in the supporting information which corroborates the claim made in the transactional document? But secondly, is it the most appropriate document, right? Um, is it the source of truth? Is it the provenance for this claim? 

And I’ll give you an example just to illustrate what I mean by that. So just say you’ve got a document and it’s talking about the headquarters of a… of a company. Then that address could be in numerous documents in the supporting information. It could be in a letter of correspondence, it could… it will definitely be in the annual report somewhere, but it will also be in the certificate of incorporation. And if they’ve moved, it will be also in, you know, a registered office address change of office address. So it’s obviously the last two documents which are really important, and therefore you really want the AI to select those documents and link it to that claim. So that’s just one data point, right? You’ve got hundreds if not thousands of these sort of, um, examples, and you need to be able to run these LLM ops really efficiently to be able to, as a whole, be able to accurately assess if the AI is doing the right thing. 

Anna Samborowska: You mentioned experimenting. Uh, it is not a standard approach, so you are developing your product in the iterative, experimental way. And how does it differ from this traditional development in which architecture typically is pre-designed? 

Jonathan Palter: Well, with artificial intelligence, uh, you have to lead with the AI, right? Uh, this is your unique selling point. And you could argue if you… if your AI isn’t capable, then you haven’t got a product. So the initial development phase is… is all about building the capability of the AI and just having enough scaffolding around the artificial intelligence so you can interact with it and measure the accuracy and so forth. And we call those early phases “the brain in the jar.” 

Anna Samborowska: Right, I remember… I remember these names, yeah. 

Jonathan Palter: So it was all about, um, having this brain, trying to develop it into a more mature, more capable, um, capable brain with enough business ROI. And then when you’ve got that, you can start thinking about, how… well, how can I upgrade the jar? And that could mean a number of things, right? It could mean, uh, building more enterprise features and functions so a team can cop… cooperate with one another, um, to do the verification together. Um, it could mean giving the team the confidence to upload confidential information, um, and be able to give the verification platform access to that so they can perform that verification. Or it could mean just the scalability aspects of it and make sure that all the services that comprise the Verify platform can scale on demand based on the user base. Um, so all of that… all that above applies, right? 

Um, and it’s so important going through these experimental phases that all the stakeholders of the project are in aligned in that it’s very unlikely that you’re going to get this linear, uh, progress. It’s going to be a lot more episodic, right? You’re going to be running experiments and it might not yield much, uh, but then you’ll hit some rich vein of, uh, fruitful experiments and you’ll get significant uplift. And so, yeah, you’ve just got to be cognizant that there… this is going to be a very episodic development process. 

And with that, you’ve got to think about, how do I QA this artificial intelligence, right? So this is not like, uh, traditional software engineering where you’ve got a very hard yes/no, pass/fail criterion. You know, the traditional software engineering is pretty much deterministic on the main and it’s rule-based. Whereas with AI, it’s very probabilistic. So you’re doing it via a data-driven way, right? So in… in that regard, you got to treat the outcomes really as a distribution, okay? And then you could look at that distribution and go, “Okay, what does good look like? Where do I set the threshold?” And you got to then apply statistical metrics, right? Whether it’s accuracy, you know, recall, precision, whatever it is… the task at hand. Uh, you got to set what good looks like and, uh, see if you can attain that with the artificial intelligence. So it’s really important to bake in that experimental element into your development pipeline, into your release schedule. 

Anna Samborowska: It sounds really interesting and at the same time demanding. However, you know, I think the most important… the biggest question is, what is your opinion? Is it really worth investing in artificial intelligence then? 

Jonathan Palter: Oh, I mean, absolutely. Okay, there’s a professor at Harvard Business School, he’s called Karim Lakani, and he came up with a really good analogy. He said in the 1990s that the internet lowered the cost of information transmission, okay? And he’s saying, like, AI will lower the cost of cognition. So that pattern recognition, prediction, ideation, creation, summarization—you name it—the things that humans have been doing forever, right, is going to be done by AI in a much more efficient and a lower cost, right? So if you take that thesis—and I got no doubt it will hold—um, it’s, you know… it… it forms a pretty strong foundational understanding why companies have to adopt AI right now. This is not an option; this is imperative. 

You know, I see AI as a foundational utility, a little bit like electricity. Um, so I don’t treat it as a tool or bolt-on. It’s more infrastructural than that, and it will permeate every area of your business. And I think any company that treats AI that way are going to see a step change in their efficiencies in the years to come. And we’re already seeing it. We’re seeing the likes of Klarna, the shopping payment provider, they adopted… they were an early adopter of AI and they’ve seen efficiency gains of the tune of, you know, $40 million, um, since they adopted artificial intelligence. So if you just multiply that out to the types of industries which use human cognition—you know, whether it’s legal like what we’re looking at currently, or whether it’s media, whether it’s medicine, logistics, you name it—you can just see the scale of the disruption which is coming our way. Um, so yeah, this is just showing you that AI, uh, is… is here and it’s going to get stronger and better and you… you need to adopt it. 

So I think that gives you effective… effectively the “why” of it’s important to adopt artificial intelligence. But I think it’s a bit more nuanced when you talk about the “how.” And I see sort of two layers to this. One is, um, more of a long-term strategic bet on the trajectory of artificial intelligence, and you want to transform your business end-to-end using artificial intelligence to get that massive uplift and differentiation. Um, but you’ve got one foot in the future, right? Um, you can’t do that now with the current capability of artificial intelligence, but assuming the trajectory holds, you’ll be able to do that in the future. But you want to start now because you want to get a head… head start and you get that compounding effect and that reinforcement loop that these sort of exponential age technologies tend to give you. 

But I think there’s a… another layer, uh, to this, which is more tactical. And that’s really grounded in use cases such as verification, like what we do, or it could be smart scheduling, or it could just be some sort of task automation. And that will give you some validation that, uh, you can get efficiency gains, you can reduce friction, and it actually just goes to show you, “Hey, AI works in real life in the enterprise.” And I think it also gives you, uh, two other things. Uh, it provides an internal cultural and educational element to it. So it gives you the ability and the opportunity to get your teams within your company used to working with artificial intelligence, trusting the outcomes that artificial intelligence provides. And also gives you a data point on validation that this AI tech works, and gives you the confidence to say, “Okay, how can we improve and increase the footprint of the artificial intelligence? Maybe start integrating with other systems, uh, in our… in our company.” So I think the trick is layering those two approaches. So, um, the… that more grounded use case one gives you that validation point, it de-risks… de-risks it somewhat, and you’re prepping the groundwork, right, for the longer-term strategic play, uh, which will hopefully give you those exponential returns. 

Anna Samborowska: It was actually a very, uh, deep insight into the artificial intelligence. Thank you. And you know, from what you just said, uh, I see—I’m not suspecting that I can see—that you are a very keen AI enthusiast. Are you using it every day, on a daily basis? 

Jonathan Palter: Oh yeah, absolutely. I mean, I use it day… every day and for different things, whether it’s, um, doing research, strategic thinking, uh, any code I’ve written—getting a… a fresh pair of eyes looking at it, an LLM looking at it for like code reviews and suggestions. But if you just roll up all those different types of, um, ways of interact with it, I think it’s just a great way to amplify your own thinking. 

There’s a guy called David Mutton who write… writes think pieces for a newsletter called “New World Same Human,” and big fan of his. And it’s all about how exponential age technologies can disrupt humans and economic… well, socioeconomic models in the future. And he coined the phrase “perspective engines” for LLMs. And I think it’s a fantastic way to think about them. They’re just not, uh, got the capability to synthesize data that it finds on the web or the data that’s been used to train the model in the first place. They’ve got this emergent capability to provide a fresh mental model that maybe you don’t, uh, or you don’t have familiarity with, or it’s reframing a subject that you don’t have currently. And you may have got there on your own volition, okay? But it could have took you hours, could have took you days, right? But this is a hack. You can… you can get to that point in minutes. So it’s an extremely valuable productivity, uh, game and it’s like jet fuel for your productivity. So anyone not using AI in their day-to-day is missing a trick. 

And there’s a perspective that’s been, uh, bandied about by experts over the years which is, you know, “AI won’t replace humans, but humans using AI will replace humans not using AI.” Okay? So I think that’s really important to have in the front and center of your mind. So as a… as a knowledge worker in your respective field, says, “Well, how can I stay relevant? How can I stay top of my game?” Um, and therefore you’ve really got to be looking at how can I integrate with these tools so I can stay relevant in this new era of AI. 

Anna Samborowska: I really like what you said, that, uh, AI is not going to replace people but people using AI are going to replace those who don’t. So John, to wrap it up, are there any final thoughts you would like to share? 

Jonathan Palter: No, I just say it’s been a… a great journey. I mean, when we first started, uh, working together, um, the landscape was slightly different than it is now, right? You know, the sort of OpenAI models and the other closed source models were not really there to be used in the enterprise domain. But obviously, the technologies matured to the point where, you know, it’s now, uh, readily available and, yeah, yeah, I can’t wait to see what we can do with this sort of technology, uh, in the… in the months and years to [come]. 

Anna Samborowska: I think we all do, we all do. And John, thank you very much for this discussion today. And thank you everyone who decided to tune in during this discussion. See you next time, stay tuned and curious.