“Unexplainable” Intelligence
Some AI is believed to be unexplainable, including generative AI (Gen AI). To accomplish a task, Gen AI can instantly pull from billions of data points, and while it’s theoretically possible to understand this process, the complexity and speed involved makes this quite daunting. This is why AI has, in some circumstances, functioned like a black box—it produced results without explaining the how’s and why’s.
Initially, when users were solely concerned with the accuracy of predictions, the need for explainability didn’t seem pressing. But as AI advances and permeates various sectors, including ones with strict regulatory requirements such as financial services, the need to understand AI’s inner workings has grown immensely.
The Need for Transparency
“Explainability in AI is similar to the transparency required in traditional banking models—both center on clear communication of inputs and outputs,” says Chris Gufford, Executive Director – Commercial Lending at nCino. “Within the model development cycle and data interpretation, explainability is essential for maintaining trust and understanding. At its heart, explainability is about achieving this transparency, regardless of the advanced nature of the AI or the mathematical complexity of the models.”
The demand for explainability only grows as predictive AI becomes more sophisticated. Explainability fosters trust and accountability in AI systems, enhances regulatory compliance, and facilitates model improvement and optimization over time. It empowers stakeholders to make informed decisions based on AI outputs and creates greater understanding and acceptance of AI technology and various domains.
Human Relevance to AI
One of the ways companies like nCino ensure explainability is through a concept called “human in the loop.” The concept facilitates ongoing model optimization and refinement, and integrating this concept is crucial for model comprehension and optimization. It involves incorporating human expertise throughout the AI model development, deployment, and execution. Human experts, such as domain specialists or risk analysts, can provide valuable human insights into the data, model assumptions, and business context, which are essential for understanding and interpreting the model’s outputs.
In the model development phase, human experts can collaborate with data scientists to select relevant features, define appropriate model constraints, and validate the model’s performance against real-world scenarios. During model execution, human experts can monitor the model’s behavior and intervene when necessary to address issues such as bias, fairness, or ethical concerns.
As we enter an exciting era where innovation meets efficiency, it is crucial for financial institutions to place explainability and interpretability at the core of their AI models to prioritize transparency, accountability, and human relevance. To learn more about the impact of AI explainability and transparency for financial institutions, download our full white paper now.
