How AI Observability Ensures Success at Scale; Insights from Venky Veeraraghavan, CPO at DataRobot

Ensuring the success of AI implementations involves more than just deploying models; it requires continuous monitoring and management of these models to detect and prevent unexpected behaviors. AI observability is a critical component in this process, allowing organizations to gain insights into how their AI systems operate in real-time. By identifying and addressing issues proactively, AI observability helps maintain the reliability and trustworthiness of AI applications at scale. This week, we have Venky Veeraraghavan , Chief Product Officer at DataRobot who leads the Product Team and is instrumental in shaping and implementing their AI platform. With more than twenty-five years of experience in product leadership, including positions at Microsoft and Trilogy, Venky has dedicated over a decade to developing hyperscale BigData and AI platforms for major global organizations. His dedication to AI observability reflects a deep commitment to enhancing the reliability and value of AI systems, fostering innovation and building trust in AI solutions. AIM Media House: Why are you particularly passionate about AI observability and predicting unexpected behaviors at scale? What inspired you to focus on this topic? “The question is, how is this useful for others? What else can we do to add value for customers with AI?” Venky Veeraraghavan: DataRobot has been building an AI platform for a long time. We started with AutoML, then moved on to MLOps. When Gen AI came along it was clear that we weren't going to be the team building large models. The question is, how is this useful for others? What else can we do to add value for customers with AI? It's evident that the initial models, and even those today, are not fully reliable. They often provide unexpected answers, leading to a confidence problem among users. People are hesitant to rely on these outputs for their business workflows. The basic response was, "I don't trust it." So, what can we do to support that trust? That's where we focused our efforts. We identified two key problems, the second one being more prominent today. First, how do you select the right LLM (Large Language Model), prompting method, and database from the hundreds available? Second, once these are in production, how do you ensure they work as expected? If they're not working, what actions should be taken to mitigate issues? These questions are very important to customers. We've dedicated a significant amount of our development hours to creating robust solutions in this area. AIM Media House: How has the role of AI observability evolved with the changing landscape of AI implementation? What significance does it hold across various platforms today? “ The emergence of generative AI has dramatically changed this landscape.” Venky Veeraraghavan: I got into machine learning (ML) about eight years ago, starting with building the internal platform for the Office team at Microsoft. The machine learning models were deployed for the homepage, relevance, and ads, making MLOps crucial. Although we didn't call it MLOps back then, it was all about ensuring that the model functioned effectively. We focused on both operational aspects, such as latency and cost, and ML metrics, such as data drift. The core of observability was to make sure the model worked correctly. If it drifted, we would retrain the pipeline with new data and redeploy it. This was the challenging part of the observability problem. However, the emergence of generative AI has dramatically changed this landscape. In traditional ML, you might look at any one prediction or a set of predictions over time to compare distributions and decide if action was needed. This process was somewhat offline. With generative AI, you're interacting with a chatbot or solution, and each input could produce a bad answer. Now, you need to understand every prediction and every generation in real-time. You must recognize when something is wrong and mitigate it immediately, as a human is on the other side of the conversation. This shift has transformed observability from an esoteric concept understood mainly by data scientists to a critical, real-time issue. No one wants to provide bad answers or data to users or face embarrassment due to poor model performance. As a result, interest in observability has surged, highlighting its importance like never before. AIM Media House: Can you shed some light on the concept of custom metrics in AI observability? What do custom metrics mean, and what significance do they hold in today's landscape of thorough supervision across various models? “When we talk about driving customer metrics, it’s important to recognize that AI doesn't exist in isolation.” Venky Veeraraghavan: I love the idea of customer metrics. We have a very strong product in that area. When we talk about driving customer metrics, it’s important to recognize that AI doesn't exist in isolation. Traditional metrics, like operational latency, count, cost

How AI Observability Ensures Success at Scale; Insights from Venky Veeraraghavan, CPO at DataRobot

Keep reading. It's free.

Related stories