GitHub
An example of how a user would interact with GitHub's genAI-powered Octobot tool.
"And, most interestingly, customer satisfaction...went up 12 points. So, now it’s at 98% customer satisfaction. I didn’t have to teach anyone. We didn’t have to roll out this technology in a way that you normally would in a corporate setting. We kind of just turned it on in a place where everyone was already asking for help.
"That’s been the big secret, I think, that when I talk to peers in other companies about how you can remove actual toil and save people’s time every day without introducing net-new behaviors? This is one example.
"Now we’re taking that process and testing about six to seven other tools currently inside the company. About 10% of our employee base is in some sort of AI tool trial, where we’re doing the same process: how do we do it without enablement? Can we put it in flow? Then we just measure the time saved on the other side.
"If it saves time and the tool pays for itself — onward and upward. What we don’t do is have a tool where you have to learn about it, get trained on it, or it’s very open ended. Like, this tool can do everything for you! Because we do find it actually makes people more productive, happier and ultimately removes their toil."
We’re trying out other tools that are giving back employees potentially 30 minutes to an hour every day. Then they get to do the work that’s more strategic for our customers and move more quickly.
Are you considering other genAI tools? "In terms of GitHub..., we’ve taken a lot of the underlying Copilot technology and we’ve applied it to our support use cases. So, when you’re on GitHub and you have a problem and have to enter a ticket, it’s very similar to how we’ve learned from our IT experience. When you start to go through the flow, and we do the normal process of how we go through the problem, there’s a stage there where you can end up talking to Copilot. And we call it Support Copilot.
"It basically is able to walk it through with the context of what you’re asking and what access you have, potentially solving your problem right there.
"On the other side of it, looking externally, we’ve also been going past the idea that Copilot is helping you write code and helping you summarize a pull request.
"What happens when you sort of flip the model on its head and you start describing what problem you’re trying to solve to Copilot? That’s the exploration we’re working on, that we’re calling Copilot Workspace. Essentially, instead of starting right in the code, you start with a problem; you start with a GitHub issue describing what you’re trying to accomplish. Copilot can take that and go from prose [written prompts] of what you’re trying to do, into the code, then you can edit the prose and it'll change the code. Then you can edit the code if you need to. Ultimately, it can test that, run that, build that and you can deploy it all starting from natural language.
"That’s an area we’re trying to push past and figure out how to write more code faster, more accurately. Sometimes the best code is no code at all and it’s just using human language."
How do you determine ROI, and have you figured out an ROI on this technology? "You can measure a million things [internally]; that’s true for software development, too. In software development, there are practices where you can measure DORA metrics — this idea of how much code you're pushing through your pipeline. You can measure space metrics. There are all kinds of methods here.
"Internally, when we were trying to work on these IT use cases, it was the same thing. How many tickets are deflected? How many tickets were closed? At the end of the day, we’re all measuring activity. Ultimately, what we’re trying to measure is time. How much time are you getting back?
"For me, when we’re doing all these measures, the ROI is baked in because we’re taking time back that previously employees were using to solve those tickets, do manual flows, write code — whatever that is. So, we take that time and invest it back into something that’s more strategic. So, the ROI has been pretty clear on this OctoBot idea; the 55% more productivity ultimately gets reinvested automatically because you’re able to move more quickly as a software developer, and then we’re trying out other tools that are giving back employees, potentially, 30 minutes to an hour every day. Then they get to do the work that’s more strategic for our customers and move more quickly.
"I think you can see that at GitHub how it feels like we’re able to do so much more than before, and it’s not because we’ve grown 50% in the last year; it’s because we’re able to invest that time back.
"So, for me I feel like every measure the industry tries to put on AI is us just trying to fit our scribes into a printing press modality and instead we all can agree if I had an hour every day where I could do something more that I love to do, that’s an easy measure. I can take that and invest it into what’s more strategic to me at GitHub."
What tips do you have for companies considering using or scaling their use of genAI? "The best thing companies can do to get started is find methods of folding AI into existing workflows — teams don’t need to reinvent the wheel, as it is easier to use AI functionalities that keeps you in the flow, not that requires net-new behavior. Meet employees where they are, give them the AI tools they need, and eventually they’ll evangelize it to others.
Some other tips stemming from my experience at GitHub include:
- Make a plan you can repeat. This isn’t the era of 'move fast and break thing,' but it’s also not the time to develop a six-month, one-size-fits-all rollout strategy. But, we can learn from decades of writing software that iterating is more important than an enormous rewrite project that you hope is correct. It’s important to pilot often, early, in small groups. You need to make experimentation a core part of your culture. At GitHub, this looks like more than 10% of our company being involved in AI pilots.
- Let AI take your toil. The best AI tools are the ones that let our best human qualities shine. We don’t need AI to be creative while we toil to manipulate the AI to do what we want. Focus on what distracts employees from doing their jobs — like calendar management for example — and let AI take the toil so your employees can unleash their creativity in bigger, more profound ways.