It's very expensive to create and train up an LLM. I’ve heard figures as high as $5 million. What have you found? "That’s the value. It’s part of the reason security is so important. The cost and the energy required to train a model is significant, but at the end of it, the output of it fits on a USB drive. So, you’ve created a perfect incentive for industrial espionage; ...these will become crown-jewel data for companies that need to be protected. If you’re an adversary, you’re going to think, ‘I’m not going to spend millions of dollars to build out my own cluster and train it and come up with a model that takes me six months. I’m just going to steal it.’
"It's not a huge amount of data. It’s 50GB. Even from an energy footprint, when we used to build even a dense data center, you’d put maybe 10kw into a rack and maybe at the high end for a really high-end, dense compute cluster, you’d go 30kw to 40kw. AI clusters need something like 100kw per rack. So, there is significant power and infrastructure required for building out these clusters."
What is Retrieval Augmentation Generation? What’s its importance? "It’s a way to bring into an LLM, without needing to do refining and tuning, access to your proprietary information where that doesn’t become part of the public training set. That’s the value. You can imagine a scenario where there’s a topic search for a call center. ...Can I get the benefit of an LLM for mastery of language, but marry that with what I know about these products that sometimes cause problems for people."
There’s a dearth of talent around AI model creation for training LLMs, prompt engineering, and general knowledge of AI. How are you addressing training for Cisco’s workforce? "Everyone is required to take our responsible AI training. It’s new, so you’re not going to find lots of people in the workforce that have all these skills that are only now being refined. So, you’re just going to have to grow a lot of this talent. And to do that, the best way to learn is through experience.
"...There’s high value in making these tools available to people in a safe way and providing an environment where people can experiment. That includes the physical as well as the software. We need to build these environments so that engineers can get their hands and feet in the data centers to understand how this technology really works. And then make those clusters and the LLMs that will run on them available to our product development teams, our software developers, and others in the company so they can get real-world experience with prompt engineering, but also software development and embedding AI features into the products. It is new, so there’s a learning curve for everyone.
"It's probably the rule of thirds where a third of people are going to struggle, a third will excel and a third will need training and help getting there."
So, how has Cisco addressed internal training? Have you created sandboxes and video classes? "Yes and yes. We’ve created the sandboxes where people can experiment and learn on the job. I think the best way to learn something is by solving a difficult problem. ...Having a real problem that you’re trying to solve is a great forcing function to focus and narrow the effort in getting real experience.
"Solve a real problem and a business outcome. We also work with product teams across Cisco — people who’ve had years of experience working with AI. Our collaboration products use a lot of AI to do things like background noise cancellation and determining who the active speaker in a room is and what’s the best camera angle in a conference room for a meeting, and summarizing the meetings.
"So, we learn from each other. Then there are certain jobs we continue to always need more of: data, design, user experience, AI, enterprise architecture. Those disciplines are always in high demand and now more than ever as everybody moves toward experience-led IT, experience-led products and having a singular user experience throughout all our products.
"AI has been around for a while. Why did ChatGPT catch on so quickly? It’s because they created a very simple interface for it instead of an API. You can go to a website and chat with this thing."