Follow ZDNET: Add us as a preferred source on Google.
ZDNET’s key takeaways
- Moving AI agents into production can be a tough ask.
- Smart professionals focus on governance and frameworks.
- They work with experts and ensure clear outcomes are set.
There’s a lot of hype about the potential of AI agents, but less evidence that the tools are producing a return on investment.
Tech analyst Gartner recently predicted that 40% of enterprises will demote or decommission autonomous AI agents by 2027 due to governance gaps that are only identified after incidents occur when these agents are in production.
Also: AI is causing cognitive fatigue. Here’s how to work with more haste and less speed
At the recent Snowflake Summit in San Francisco, three digital leaders explained how their organizations put agents into production. They shared three lessons for other professionals looking to exploit AI: use frameworks, exploit experts, and monetize data.
1. Focus on frameworks
Matt Luizzi, VP of analytics at wearable technology specialist Whoop, said his organization collects biometric data 24/7 to power its health and wellness insights, with Snowflake supporting the firm’s internal analytics services.
Luizzi said agents play an increasingly important role in this process, particularly Snowflake CoCo, the technology specialist’s coding agent for developers and data engineers.
“We’ve been using CoCo for several months now, and started with just the analytics team, which is people who could quickly look at a query response and say this is correct or not, and trying to figure out how to scale that process out,” he said.
“Now we’re at the point where we’ve created more formalized evaluation frameworks and are starting to roll agents out at scale.”
Also: Forget productivity: Here are 5 strategic shifts that drive real AI value
Luizzi said the firm has software engineers who deploy A/B tests and use CoCo to analyze the results, propose the next feature, test it, and iterate.
“This approach is rapidly accelerating the way that we’re shipping not only business value, by automating the experimentation framework, but also the customer value,” he said.
Luizzi said his organization was fortunate that the underlying plumbing was already in place for its agentic explorations, due to the firm’s data being centralized on the Snowflake platform. They used the firm’s Cortex AI service to start testing agents and learning lessons.
“We learned fast that context was everything,” he said. “That meant really leaning into the semantic layer and making sure the context is in a structured place.”
Also: Why I ditched Copilot for Claude in Word, Excel, and PowerPoint – and how you can, too
Luizzi said a key lesson is that frameworks are crucial to successful agentic AI explorations.
“We’re trying to do everything in a more repeatable manner, the same way that we’ve done with our data architecture for the past 10 years,” he said. “Building repeatable frameworks that enable us to scale these AI workloads is something that we’re taking forward with us.”
2. Use expert analysts
Madeleine Want, VP of data at sports specialist Fanatics, manages data engineering, data science, and machine learning across the organization’s betting and gaming division, with this activity supported by the Snowflake platform.
“When we began experimenting, we weren’t sure what would stick and what would slip, but we found that what stuck was the better the condition of the underlying data and the better the governance of it, the more easily the LLM was able to derive meaning and answer questions effectively,” she said.
While that focus on data and governance might sound like an obvious thing to say, said Want, it certainly wasn’t the case 18 months ago.
“We had a lot of experience as an organization building bespoke machine learning models, so it was hard to believe the idea of importing a third-party model and just plopping it right on top of the data could work for analysis. But now that approach is very much embedded in the way that we do everything,” she said, before outlining how her organization moved from exploration to exploitation.
“We had success early on in the domains that were well bounded in context, and where we had expert analysts who understood the business domain top to bottom and were able to coach the agent.”
Want said her organization has scored more successes over time. The investment they need to make in the context layer is decreasing, as is the degree of supervision an agent requires before it can start answering questions autonomously.
“Our ability to measure the accuracy of the answers is increasing, because we’re now introducing scaled evaluation frameworks, which are helping us have confidence in how agents are answering when we’re not looking, which is kind of the whole point.”
Also: Worried AI agents will replace you? 5 ways you can turn anxiety into action at work
Want said these successes mean the scope of agents is increasing. Rather than just being limited to analytics, other professionals see the positives and want to explore agents.
While Fanatics still uses Snowflake’s interfaces and agents, the company is embedding APIs and responses into other third-party tools so people can do more with data-powered insights.
“Users want to go further and do more with operational use cases,” she said. “People are demanding to be able to access those insights through a variety of different channels and consumption mediums, because they need to be able to use data where they’re working.”
3. Monetize your data
Sriram Sitaraman, CIO at software specialist Synopsys, said his organization is a long-time Snowflake customer that uses the data platform and its agentic services, such as CoCo, to power its decision-making processes.
About 18 months ago, Sitaraman said the company recognized the potential for AI agents to fulfill the tasks of junior employees, such as running quick queries, creating graphs, and deriving insights.
“We took advantage of that capability, and we said, ‘OK, look, if we create a knowledge agent, we can start deploying it in multiple dimensions.'”
Also: 90% of AI projects fail – here are 3 ways to ensure yours doesn’t
Examples include a revenue agent for the finance department that runs reports and a debug agent for the ticketing system associated with the firm’s data centers.
Sitaraman said the team assessed the potential of AI across three dimensions: the quality of results, time to results, and cost of results.
They discovered AI has a positive impact in all three areas, which he said is a significant breakthrough: “In the past, you had to sacrifice one or the other.”
Now, rather than having to reprogram systems each time the AI model is tweaked for context, it’s possible to focus on insights rather than worry about underlying concerns.
“Start with data — monetize your data using AI,” said Sitaraman, reflecting on his firm’s agentic journey. “It doesn’t matter how much volume you throw at the initiative, because AI is just truly a linear scale. The more data AI has, the better decisions it makes.”
Also: 5 ways to use AI when your budget is tight
However, Sitaraman also issued a warning. “One thing we realized is there’s not a lot of difference today between automation and autonomy, and so you have to be careful,” he said.
“Do you want to automate a process or do you want to actually create an agent, which involves a different cost structure, usage pattern, and governance?”
Sitaraman encouraged professionals to identify the right use cases, build the right frameworks, and never underestimate what an agent can do.
“You can roll out an agent and say, ‘This is a sales ops agent.’ Often, there’s nothing to stop it from also becoming a sales analyst agent or another type of agent,” he said.
“So, it’s important to ask, ‘Is this what we want it to do?’ Frameworks are very important, as are skills. You need to think the process through carefully.”
