Smart Pilots Turn Public-Sector AI Into A Defensible Line Item

Public-sector AI conversations tend to fixate on ambition: whether agencies are being bold enough, moving fast enough, doing more than chatbots. The better question reframes the problem entirely. The chatbot is where public-sector AI starts, not because agencies lack imagination, but because regulation, labor considerations, and compliance make everything slower, and pilots are how the case gets built. What actually determines whether AI goes further depends on whether the data underneath it is clean, whether the token consumption is budgeted like the real operating cost it is, and whether the IT team has been rebuilt around the skills AI demands. Get those three right and the ambition takes care of itself.

Puneet Sharma is Chief Information Officer at the Boston Public Health Commission, the oldest health department in the country. He owns a full technology portfolio spanning AI, infrastructure, enterprise applications, data and analytics, information security, and digital across eight business units, and has put multilingual NLP, demand forecasting, and risk-triage AI into production. He also serves as adjunct computer science faculty at CSU Channel Islands. Sharma's recent report outlines six priorities that will separate the institutions that get real value from technology from the ones that spend another year piloting, and his read is that the public sector's caution has nothing to do with nerve. It's structural, and the constraints are real.

"Private sector is the first to adopt, first to implement. The public sector, higher ed, healthcare, they're a hundred steps behind, because the whole organization hierarchy is different," he said. Between chain-of-command approvals, labor-union considerations when a deployment touches jobs, and compliance regimes like HIPAA and FERPA, agencies begin with the low-hanging fruit because it's the only thing that clears the bar quickly, which is why transformation in the public sector is as much a service-design and organizational challenge as a technical one.

• Put AI on the balance sheet: Sharma advises treating AI as a budgeted operating cost rather than a line-item afterthought, and the meter that matters is tokens. "That's how these LLM companies make money, on tokens. As an agency, you have to make sure you have enough funds to pay for those tokens budgeted," he said. Budgeting AI well requires matching the model to the job, an economic discipline that rewards the same workflow-first, business-target thinking CIOs apply to any major platform decision.

• Right-size the model to the task: That matching is where Sharma gets specific about not overpaying. Using the most powerful model for everything is how a token budget evaporates. "It depends how complicated the task is. For accounting you can use one model, but for research work you want a different one." He recommends first-year programs budget a genuine floor of roughly $150,000 to $200,000, pilot with 50 to 100 people, then scale as the value proves out, rather than committing enterprise-wide before the economics are understood.

Underneath the budget sits the technical reality that AI is a catchy label for a great deal of unglamorous engineering. "Everybody loves artificial intelligence, but it's complicated to run. First you have to have a data repository, a data lake or warehouse. Then you have to make sure that data is correct," Sharma said. Only then come the connective layers, the MCP middleware governing which user has access to what, and the APIs that increasingly define how enterprises coordinate AI against their real data and systems.

• The failure that starts with data: The foundational risk Sharma worries about most, from direct experience, is data quality, because AI faithfully reproduces whatever flaws it inherits. He described pulling terabytes of data from a third-party source his team assumed was clean. "We connected the API and started running queries, and we were getting all incorrect outputs. We started thinking, 'Why is this happening?' Then we had to go back and fix it." The lesson is that a data-engineering function that curates and QA's every pipeline from source to a trusted, curated bucket is the precondition for any AI output an agency can act on.

• Compliance sets the pace: When the data feeds a public agency, the cost of getting it wrong climbs from a bad report to a legal liability. Sharma points to the everyday example of meeting transcription. "There are a lot of lawsuits in public agencies. 'I didn't say that, but the transcript says that. The AI took its own meaning,'" he explained. That risk profile forces a higher bar of proof before anything scales, which is why the pilot phase is the mechanism that makes adoption defensible.

Sharma's sharpest claim is about people, and it comes from watching both his own portfolio and his students' job market. He believes the enterprise-application department has about three years left in its current form. As faculty, he sees the demand signal directly. "Recruiting companies tell me they don't want software engineers anymore. They say, 'You're wasting time. We want AI developers.'" The two roles he considers non-negotiable, data engineers who guarantee the data is correct and AI developers who can train and tune models, are the minimum staffing before any serious AI project can move forward.

Ultimately, he says, public-sector AI doesn't stall on vision. It stalls when leaders skip the foundation, and the CIOs who succeed are the ones who fund the tokens honestly, curate the data ruthlessly, and rebuild the team deliberately before chasing the ambition everyone else is talking about.