Trust is a major issue hampering the adoption of AI technology, and at the core of this is the problem of ‘black box’ AI.
As enterprises race to implement AI projects, many have bought-in pre-trained black box models, which provide answers, but no explanations.
Many of the better-known AI tools are considered black boxes – we cannot see inside, they perform tasks well, but we’re not necessarily sure how.
VP & Country Manager UK&I, Snowflake.
40% of organizations called out explainability as a blocker preventing trust in AI, according to McKinsey. To truly derive value from AI technology in the enterprise we need something different from the existing approach many have taken. We need solutions that offer greater transparency in how they work and how they arrive at the answers they deliver.
If CIOs, developers and end users can work together, we can arrive at AI that is transparent, adaptable and aligned with real needs. Just as we expect academic researchers to ‘show their workings’ by citing their sources, we should expect the same of AI systems.
New technologies and methods already exist to enable this accountability, and regulations help mandate transparency and accountability. Businesses need to engage proactively with this: an AI system is in many ways like an employee.
We ‘hire’ it to do a job, and, like an employee, we need to understand what it does in order to measure and monitor its performance.
The cracks in the first wave
When ChatGPT changed the world forever back in November 2022, the barrier to entry of training capable models meant enterprises mostly relied on pre-trained, proprietary options from large technology companies to move quickly.
Many have continued this strategy, even though these models are not trained on a business’ data, often failing to meet the specific needs of that organisation, and offer little transparency about what is going on ‘under the bonnet’.
In conversations with IT decision-makers, we hear genuine issues raised around AI compliance, ethics and competitiveness. Potential toxic biases and discrimination can have real world impact for people interacting with and depending on the output of AI-powered tools. Ultimately, these risks slow down adoption and limit potential use cases and applications.
For enterprises, this is a key risk around AI, and a barrier towards building trust. The same McKinsey survey found just 17% of respondents said they were currently working to mitigate issues around explainability, despite acknowledging it as a problem.
The report also said that systems which offer more explainability will be a key enabler as AI moves beyond early use case deployments to scaled adoption across the enterprise. Even though regulation has made forward steps, risk management still needs to come from businesses taking a lead in tackling this issue themselves.
The shift toward openness and customization
As businesses begin to step back and take a longer view on AI adoption, it is becoming clear that transparency and governance are just as important as speed. For years, adopting AI meant relying on opaque, one-size-fits-all models where you had little visibility into how decisions were made and even less ability to tailor them to your business.
These “black box” systems forced organizations to accept generic outputs, release their proprietary data to third-party providers, and hope the results were accurate and unbiased. This can be a significant blocker for many highly-regulated industries and use cases.
That approach is rapidly giving way to a new paradigm: open, customizable models that give businesses full control over how AI is built, fine-tuned, and deployed using their own data, on their own terms.
By opting for a data and AI platform that brings compute to the data, enterprises can ensure that AI works within governed parameters and you have model choice and control. Your data never has to leave your account, so you maintain governance, privacy, and compliance throughout.
The result is AI that actually understands your business context, delivers more relevant answers, and evolves as your needs change. This puts the strategic advantage of AI firmly in your hands, not a vendor’s.
Building AI that people trust
As AI moves from experimentation to real business decisions, trust becomes the defining challenge. Agents – AI systems that can reason, take actions, and work across tools on your behalf – represent an enormous leap in productivity, but only if people trust them enough to rely on them.
That trust doesn’t come from marketing promises; it comes from being able to see exactly what an agent did, why it did it, and what data it used to get there – while using a human in the loop for key decisions. Observability and transparency aren’t just technical nice-to-haves, they are the foundation that determines whether your organisation will actually adopt AI or keep it sidelined as a pilot project.
Explainable and observable AI are key parts of our Responsible AI principles at Snowflake, and it needs to be a governing principle for any organisation that hopes to reap the benefits of AI. Compliance, legal, and business teams need to be able to verify AI behavior just as rigorously as any other business process, turning AI from a black box into something your entire organisation can confidently stand behind.
Taking an integrated approach involving CIOs, developers and data platforms, companies can work to build AI that people find easy to trust, and which also delivers for the enterprise in a way that pre-built, opaque AI cannot.
Opening the box
In the rush to capture the benefits of AI, many enterprises have discovered that black box systems, however powerful, introduce risks that outweigh their initial convenience. Lasting value will only emerge when businesses can understand, interrogate and confidently govern the technology they deploy.
The path forward lies in transparent, explainable AI that is observable, accountable and aligned to an organization’s own data and needs.
By embracing openness, integrating human oversight and adopting platforms that prioritize responsible, well-governed AI development, enterprises can finally unlock AI’s potential at scale.
This is not just a technical preference but a strategic imperative, as transparency will determine long-term trust, regulatory readiness and competitive advantage.
Organizations that invest now in explainability and observability will be best positioned to build AI systems that deliver dependable value long after the first wave of black box hype fades.
We list the best Robotic Process Automation (RPA) software.
This article was produced as part of TechRadar Pro Perspectives, our channel to feature the best and brightest minds in the technology industry today.
The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/pro/perspectives-how-to-submit
https://cdn.mos.cms.futurecdn.net/Thi6y93AMWrCXJAEiHDQbL-2560-80.jpg
Source link
