Need a Coder? ChatGPT Can Do the Job. Mostly

If you're not already using AI chatbots at work, there's a ton of evidence to show you're missing out on increased efficiency and productivity. One of the smartest ways AI can help your company is with coding tasks. Trained on billions of lines of existing software code, AIs like ChatGPT can cover gaps in your developer team's experience, or help them solve really tricky problems. Now researchers find that ChatGPT really is successful at producing working code--but not 100 percent reliably. And it helps if that thorny coding problem your dev team is wrestling with has been tackled by other developers a few years ago. ChatGPT can code, just not as reliably as some human coders The new study examined how well ChatGPT could write code, and measured its functionality, complexity and security, reports the IEEE Spectrum news site, run by the Institute of Electrical and Electronics Engineers. Researchers found that when it came to functionality, ChatGPT could spit out working code with success rates as low as 0.66 percent or as high as 89 percent. This is a massive range of success, perhaps more than you might have thought reasonable, but as you may expect, the difficulty of the problem at hand, programming language, and other factors played a part in its success--just as is the case with human coders. That's not surprising, since generative AIs like ChatGPT work off of data that's put into them. Typically that means the AI algorithm has seen billions of lines of existing human-written code--a data repository that was built up over decades. To explain some of the variability of ChatGPT's results, researchers showed that when the AI faced "hard" coding problems it succeeded about 40 percent of the time, but it was much better at medium and easy problems--scoring 71 percent and then 89 percent reliability. In particular, the study says ChatGPT is really good at solving coding problems if they appeared on the LeetCode software platform before 2021. LeetCode is a service that helps developers prepare for coding job interviews by providing coding and algorithm problems and their solutions. A researcher involved in the study, Yutian Tang, explained to Spectrum that if coders asked ChatGPT for help on an algorithm problem set after 2021, it struggled more to produce working code, and sometimes even failed to "understand the meaning of questions, even for easy level problems." This 2021 date isn't rooted in trickiness of code problems however. Developers continually encounter coding difficulties, and it's just that some will have already been encountered and solved by people before. So the AI's coding expertise is influenced by time: A long-solved coding issue will have appeared more often in the AI's training database. Even more interestingly, the study found that when ChatGPT was asked to fix errors in its own code, it was generally pretty bad at correcting itself. In some cases, that shortcoming included putting security vulnerabilities in the code the AI model spewed out. Yet again, this is a reminder that while AIs are incredibly exciting, and can definitely provide a big boost for small companies whose coding teams may lack diverse expertise, ChatGPT isn't going to replace them anytime soon, simply because its results can't be relied on every time. Rather, an AI assist is best used as a tool that developers can consult to help their output. And all AI-generated output probably should be double-checked by human experts before it's run live--to make sure its hasn't left any security loopholes open, for example. Coders condemning ChatGTP come across a copyright snag Meanwhile, coders who sued OpenAI, Microsoft, and GitHub over the issue of AI training data suffered a setback Friday when a judge overseeing their $1 billion class-action suit dismissed their claims. The coders alleged the AI companies had "scraped" their code to train the AI algorithms without permission, violating open-source licensing agreements. They were trying to leverage the Digital Millennium Copyright Act, a law you might know about from so-called takedown notices against user-uploaded content on sites like YouTube. It's invoked when a music publisher says a publisher of web content shouldn't have used a particular track without proper permission, for example. But the ruling said their claims were without merit since they failed to show that Copilot, Microsoft's version of ChatGPT, could replicate the code "identically," Bloomberg law reports. AI critics will note a subtle issue here. Other content creators, ranging from recording labels to big-name newspapers, have pursued legal action against AI companies like OpenAI on broadly the same grounds, but generative AIs tend not to 100 percent "reproduce" data they've been trained on, simply because of the statistical nature of the way their algorithms work. Contrary to how it may sometimes appear, coding is more like an art mixed with a science, and code doesn't have to be "exact"--it can be as creative as a painted artwork or a hand-written newspaper article. Developers can use different techniques to solve the same problem, and, having been trained on lots of this sort of different code, it seems like now AIs are churning out their own solutions based on the original material. And with Microsoft's AI chief showing his cards last week, alleging that your content is fair game for AI scraping if it's ever been uploaded to the open web, it seems that this sort of AI intellectual property issue, and the lawsuits that then follow, is only going to get more complicated. Your big takeaway from this tussle: Keep your company's secrets well hidden from the internet and its hungry AI data bots. BY KIT EATON @KITEATON