Beyond AI Copilots, The Next Evolution Of Data Engineering Demands High-Level Quality Control

Enterprises are caught in a tense tug-of-war with board-level pressure to adopt generative AI and agentic workflows crashing against a disappointing reality of low ROI and widespread AI fatigue. While every vendor promises a silver-bullet solution, reports suggest that as few as 25% of AI pilots are driving even marginal returns, leaving executives to wonder if they are innovating or just keeping up with the Joneses. Often, the failure isn't in the technology, it's in the approach of culture, communication, and a lack of executive buy-in intertwined.

We spoke with Shiv Malhotra, Manager of Centralized Data Engineering at Tanium, a seasoned leader who has spent over 13 years on the front lines of data and analytics. Drawing on experience from startups to large-scale enterprises, he argued that for AI pilots and projects to succeed, organizations must fundamentally shift their perspective. Instead of chasing flashy solutions, they must start by reverse engineering the problems worth solving.

• Discovering impact: Malhotra said this isn't a new concept, but his background as a data and analytics engineer points him to just "reverse engineer" most of the challenges he encounters in his day-to-day. In Malhotra's perspective it's better to, "Start from your problem to find the solution because starting at the solution you're susceptible to creating hypothetical problems that don't even exist, aren't impacting a great enough margin from a cost-benefit, or could just be solved by getting you the data that you need."

This problem-first philosophy is a knockout punch to silver bullet syndrome—a recurring condition in tech where hype overtakes reality. Malhotra described the current AI craze as just the latest chapter in a familiar story, drawing a direct parallel to past tech cycles of the DotCom boom, Big Data, cloud computing's evolutions, and the API wave.

• The hype-cycle repeats: Any professionals who have been in the data and analytics world over the past two decades know this hype is typical for new tech. As Malhotra explained, "Before cloud, we had the same thing with big data." He recalled the era when on-premise infrastructure was treated like "one of those kids that nobody wanted to hang out with in school," only for companies to start returning now after discovering the runaway costs of their cloud-first mandates.

• The human advantage: Today's AI boom is creating similar unintended consequences. "At Tanium, we use different solutions for data engineering and each of them now have their own copilot," Malhotra shared. "The problem is when you try to analyze one copilot's score with another copilot's score, now these damn scores are not matching. So which score do you execute strategy from?" This example has been echoed by many leaders struggling to understand which signals to follow, and often leads to more confusion and disagreements in strategy meetings. "More often than not it's a forcing factor requiring human teams to step in and guess who's there to save the world?" he asked rhetorically. "Data and analytics engineers, saving the world one data set, integration, or audit at a time."