From Precision to Efficiency: A New Perspective on Data Engineering

In a captivating talk at the recent Data Engineering Summit, Sudarshan Pakrashi, the Director of Data Engineering at Zeotap, addressed the elephant in the room for many data-driven organizations: the rising costs of managing and processing ever-expanding data sets. His solution was both simple and profound, changing the perspective on how we handle voluminous data.

Painting the Data Landscape

Pakrashi painted a vivid picture of the challenge, demonstrating the magnitude with a relatable example. Suppose an organization like Zeotap tracks a million users' impressions across 100,000 ads daily, with each user assigned a unique 64-gig hash key. The amassed data would be around 160 GB daily, ballooning to 50 terabytes monthly.

This scenario represented a single use case, with actual situations involving several analytic needs and a consistently expanding data repository. The underlying problem here was the soaring storage and computation costs, proportional to the raw data size, burdening organizations financially.

Challenging the Convention

The novel approach Pakrashi proposed was based on challenging the need for absolute accuracy. In many operational contexts, like real-time alert systems or reporting dashboards, exact numbers aren't necessary. Instead, a broad understanding of patterns and trends is often sufficient. This paradigm shift opened the door to new solutions that favor speed and scale over absolute precision.