Databricks Says Clean AI Data Doesn’t Exist and Acquires Fennel

By all accounts, Databricks has a data problem and so does everyone else.

“Everybody has some data, and has an idea of what they want to do,” said Jonathan Frankle, chief AI scientist at Databricks’ Mosaic AI. “Nobody shows up with nice, clean fine-tuning data that you can stick into a prompt or an [application programming interface].” That gap between ambition and usable input has become the defining bottleneck in enterprise AI.

Dirty data, not model size or GPU availability, is what’s holding back most organizations from realizing the promise of generative AI. And Databricks, a company that’s built its reputation on infrastructure for training large models, now finds itself facing the same challenge its customers do: the data is too messy.

To close that gap, Databricks is advancing a new machine learning technique and acquiring companies that attack the issue from a different angle. Its latest training method, Test-time Adaptive Optimization (TAO), circumvents the need for hand-labeled datasets entirely. Instead of requiring curated fine-tuning data, TAO identifies the best outputs a model can generate and trains the model to prefer those, an approach known as “best-of-N.” The selection is governed by DBRM, the Databricks Reward Model, which learns from human preferences to decide which responses are worth keeping.