Beyond AI Copilots, The Next Evolution Of Data Engineering Demands High-Level Quality Control

Enterprises are caught in a tense tug-of-war with board-level pressure to adopt generative AI and agentic workflows crashing against a disappointing reality of low ROI and widespread AI fatigue. While every vendor promises a silver-bullet solution, reports suggest that as few as 25% of AI pilots are driving even marginal returns, leaving executives to wonder if they are innovating or just keeping up with the Joneses. Often, the failure isn't in the technology, it's in the approach of culture, communication, and a lack of executive buy-in intertwined.

We spoke with Shiv Malhotra, Manager of Centralized Data Engineering at Tanium, a seasoned leader who has spent over 13 years on the front lines of data and analytics. Drawing on experience from startups to large-scale enterprises, he argued that for AI pilots and projects to succeed, organizations must fundamentally shift their perspective. Instead of chasing flashy solutions, they must start by reverse engineering the problems worth solving.

• Discovering impact: Malhotra said this isn't a new concept, but his background as a data and analytics engineer points him to just "reverse engineer" most of the challenges he encounters in his day-to-day. In Malhotra's perspective it's better to, "Start from your problem to find the solution because starting at the solution you're susceptible to creating hypothetical problems that don't even exist, aren't impacting a great enough margin from a cost-benefit, or could just be solved by getting you the data that you need."

This problem-first philosophy is a knockout punch to silver bullet syndrome—a recurring condition in tech where hype overtakes reality. Malhotra described the current AI craze as just the latest chapter in a familiar story, drawing a direct parallel to past tech cycles of the DotCom boom, Big Data, cloud computing's evolutions, and the API wave.

• The hype-cycle repeats: Any professionals who have been in the data and analytics world over the past two decades know this hype is typical for new tech. As Malhotra explained, "Before cloud, we had the same thing with big data." He recalled the era when on-premise infrastructure was treated like "one of those kids that nobody wanted to hang out with in school," only for companies to start returning now after discovering the runaway costs of their cloud-first mandates.

• The human advantage: Today's AI boom is creating similar unintended consequences. "At Tanium, we use different solutions for data engineering and each of them now have their own copilot," Malhotra shared. "The problem is when you try to analyze one copilot's score with another copilot's score, now these damn scores are not matching. So which score do you execute strategy from?" This example has been echoed by many leaders struggling to understand which signals to follow, and often leads to more confusion and disagreements in strategy meetings. "More often than not it's a forcing factor requiring human teams to step in and guess who's there to save the world?" he asked rhetorically. "Data and analytics engineers, saving the world one data set, integration, or audit at a time."

The true cost of chasing silver bullets isn't just the initial spend; it's the crippling organizational debt and human resources expenditure that follows. Malhotra explained the cascading human cost of unwinding a bad technology decision.

• A vicious cycle of waste: "Once you implement a product, it is extremely hard to peel it off. You're not going to use AI to peel off a solution. You're going to talk to your IT team, your infrastructure team, your infosec team, which takes them away from other critical projects." Another sentiment shared by execs who are waiting to see what tech is truly capable of driving results so their organizations don't spend more time, money, and resources to end up back to square one.

• The solution is a slow, methodical, and human-centric approach to change. Malhotra pointed to frameworks like ADKAR—Awareness, Desire, Knowledge, Ability, and Reinforcement—as a model for managing the cultural side of transformation. To avoid the trap of top-heavy decisions, he advised leaders to bring in 'executives-minus-two,' ensuring that the individual contributors who will actually implement a solution are part of the decision-making process. This philosophy puts the focus back on the unsung heroes of AI: the data and analytics teams who form the foundation of every initiative.

• The importance of data cleaning: "Without good orchestration, governance, or pipeline handling you can't do anything meaningful with AI," Malhotra said. "If you're going to feed AI garbage, you're going to get the results that are garbage." He equated a number of early enterprise AI wins to the grunt work of data and analytics teams saying, "Their work is what allows enterprises to have all of these beautiful shiny things you can use with AI and LLMs and not the garbage you could be wading through, so without them you wouldn't have any of that."

• The magic table myth: Malhotra shared a common story that illustrates the disconnect often experienced between business units and data teams, where colleagues might have a skewed view of how data and agentic workflows actually come together. People from across an organization will come to the data group and say something to the effect of 'I talked to my peers on a completely different team and they implemented this agentic workflow for their customer success journey six months ago. Where can I get this data?'. Malhotra explained, "Nine times out of ten my response is when did they implement this because our team wasn't looped in." This story illuminates the immense fragmentation persisting across enterprises, and how easy it is for shadow AI to gain footing in an organization.

To get beyond the fragmentation and communication challenges, Malhotra argued we should be leaning into communication and transparency across organizations. From Malhotra's perspective, we have to get human-to-human communication and support locked down before looking forward and attempting to leverage human capital with AI. Once an organization has a clear framework in place, Malhotra spoke of AI not as a replacement for human work, but as a powerful tool for augmentation. In his mind, the key to generate continued leadership buy in and employee engagement is to reframe AI's role and build momentum through small, undeniable victories.

• Genius junior employees: "I tell my team and peers that you need to consider AI as really genius junior employees," Malhotra advised. "Let AI handle mundane tasks, like a product team having to browse through 200 million records of customer data to develop trends and patterns. That's the work AI should do, while humans focus on the strategic things: the end goals and the vision." Malhotra envisions a future where the engineer's role evolves. AI will provide the blueprint—for example writing 600 lines of code—and the human's job will be to orchestrate, document, set performance thresholds, and fine-tune the output. The focus shifts from manual coding to high-level architecture and quality control.

• Start small, win big: Malhotra sees the path to organic growth is a practical one once you have your foundation in place. "Pick a problem that actually exists. Put your heart, your soul, your blood, sweat, and tears into solving that problem," he urged. "Make sure that project is successful. And then you highlight the hell out of it. Highlight it in your stand-ups, in your peer reviews, and in your leadership meetings." In his experience when people see the same thing highlighted, they start connecting it to the problems they have. Malhotra explained, "From here it's a snowball effect where you can engage those folks and ask, 'Is there a big problem you have that we could solve next?'"

Ultimately, as Malhotra outlined the path to reverse engineered success, he admitted the success experienced by enterprises will be fleeting. In his view, by the time some of the late adopters implement a durable, grounded strategy that solved today's challenges, a new set of promises and pressures will inevitably emerge. "By the time we get there, something else is going to come up, and leadership will want us to do that," Malhotra concluded. "It's this ever-changing North Star that enterprises keep chasing."