Why reliable data is essential for trustworthy AI

Quality of data

The challenge is identifying where and how AI is being used in the enterprise, as well as the quality of data – especially the data used for training the models. As Gartner’s Henein suggests, AI suffers from the same data problems as any analytics system: garbage in equals garbage out.

But with AI, we are even more likely to take its outputs at face value, he said. “Humans favour suggestions from automated decision-making systems, often ignoring their own better judgement,” said Henein. “But this new generation of hallucinations, with answers that are very detailed, with references, and are extremely eloquent, pushes that automation bias to new heights.”

Much also depends on the type of decision AI is supporting, with some tools posing a greater risk to the enterprise than others.

“This is one of the hardest things,” said Tharishni Arumugam, global privacy technology and operations director at AON. “A lot of times you have people thinking, ‘I want to know about any little use of AI’. Actually, do you really need to know about a little translation service that your third party is using? Probably not, but you want to know when a third party is using your health information to provide predictive analysis to your employees. And so there is a big misunderstanding right now about what we need to know from a vendor perspective.”

This, she said, links directly into data governance, and organisations with mature data governance policies are less likely to fall foul of AI’s pitfalls.

This covers basic data quality, but also, as Gartner puts it, whether the data are both accurate and diverse enough to produce reliable results that are free from bias and hallucinations. This is sometimes termed “AI-ready data”, and Gartner warns that few organisations can really say they have that type of data – yet.

Loss of trust

The problem is made worse when organisations link AI models together across a decision-making process. As each model feeds into the next, confidence levels in the final conclusions will drop. But this might not be obvious to the user, or consumer.

“Very big models have access to insane amounts of data,” said Henein. “A lot of that data is from the internet, and we all know that the internet is not quite as curated from a quality of content perspective as you would like it to be.

“And that’s a fundamental problem,” he said. “It is at the heart of these hallucinations.”

According to Henein, models are not currently giving guidance to their accuracy, either in percentage terms or even on a simple scale such as red, amber and green. “If we had that indication of the accuracy of the response, maybe it would assuage some of the concerns around hallucinations,” he said.

Lineage of data

Confidence also means understanding the lineage of data as it moves between systems.

This includes data that moves from enterprise systems or data warehouses and lakes into AI, as well as – potentially – AI results that are used as inputs into other models, or even, potentially, to train AI. Gartner predicts that, within two years, three-quarters of companies will use GenAI to create synthetic data that, in turn, could be used to train machine learning models.

Data scientists also need to build guardrails into AI systems to reduce risk and prevent abuse of the tools.

This could include limiting or restricting the use of personal identifiable data, health information, intellectual property, or even unchecked and unqualified data sources.

“Ultimately, the data that you’re feeding into the model, the data that you’re using to train your models, is extremely important,” said Junaid Saiyed, chief technology officer at data governance and data intelligence firm Alation.

“If you don’t feed in accurate, trusted data, you’re going to get not-so-good recommendations and predictions. Whatever you’re looking to get out of your AI, whatever you’re looking to get out of your models, trusted data leads to trusted AI.

“People are looking for that confidence code,” he added. “It’s not just the final answer. They want to know the confidence along the way. What is your confidence in the data that was fed into the model, and your confidence in the model itself? You might even be okay with a less sophisticated model, if the answer is explainable.”

Building confidence

Unless chief information security officers and chief data officers can build that confidence, users will be reluctant to use AI tools, and customers will be unlikely to trust their advice or recommendations.

“In business to business, you need to provide this level of trust,” said Daniel Gallego Vico, an AI and machine learning researcher, co-founder of PrivateGPT and business AI service Zylon, and a speaker at the Tech.EU summit.

An engineer, for example, will not use an LLM’s recommendation for a design if he or she does not trust the data. “If I’m building a bridge and the bridge collapses, the lawyers will come after me, not the LLM,” he said. “I need to be sure what the LLM is producing is right.”

For Vico, however powerful the AI tool, humans have to remain part of the workflow. “You have to understand what data sources the LLM has used to generate the answer,” he said. “That way, you can double-check.”