When Erik Costlow is not working as senior director of product (cloud solutions) at Azul, he is staff editor at InfoQ. In September, Costlow moderated an InfoQ roundtable on AI integration for Java. The wide-ranging discussion touched on finding uses for AI, how it’s being used today, some best practices, and what the future might hold.
On September 18, Azul Senior Director of Product Management and InfoQ Staff Editor Erik Costlow moderated an InfoQ roundtable panel, AI Integration for Java: to the Future, from the Past. Panelists were Asir Selvasingh, principal architect for Java on Azure for Microsoft; Jonathan Schneider, co-founder and CEO of Moderne; Dov Katz, managing director and distinguished engineer for Morgan Stanley; and Svetlana Zemlyanskaya, machine learning engineer for Jetbrains.
A few highlights follow.
Use AI for boring tasks you don’t want to do
Costlow brought up using AI for maintenance. He asked Katz about some maintenance work he did and how he decided whether to use AI or allocate people to do it.
Katz noted that beside being boring, humans maintaining 19,000 files are prone to error and inconsistent work. “You start to say to yourself, we’ve got to find a better way to do this than go get a bunch of people focused on it. Inconsistently. By hand. And things like that,” he said.
Katz said he broke the problem into two categories – refactoring and replatforming. Maybe that’s upgrading dependencies.’
“The code you want to keep you tend to want to refactor,” he explained. “The code you don’t want to keep, you tend to want to replatform.”
“AI is great to get you on the right starting point,” Katz said. “What we’ve actually done is we’ve leveraged generative AI to help us understand what we were doing so that we can redo it a different way. Tell me what my requirements are from looking at my code, and now implement those requirements in a different way. It’s a great way to get started.”
However, one AI tool won’t scale to 19,000 files, and Katz recommended combining tools. He said organizations should use generative AI to imagine things they want to do and then figure out how to do them at scale.
Running the numbers based on an upgrade to Java 21 for 15,000 code bases, Katz estimated 50,000 hours of time saved, which is about 25 full-time consultants.
“That’s the boring stuff,” he said. “Imagine if there’s other stuff I actually want to do, like use AI, or any of that I need to get on these modern Javas first.”
That’s the boring stuff. Imagine if there’s other stuff I actually want to do, like use AI or any of that, I need to get on these modern Javas first.
Dov Katz
Pushing AI beyond the boring stuff
Once organizations have solved their maintenance problems and freed up time for cool features that can enhance productivity, how do they push AI to do even more? Costlow asked Zemlyanskaya about a product she worked on that uses AI on the backend. “The thing that you make available is actually AI to the engineers,” he said. “Can you talk a little bit about how integrating AI into your application works? What’s local on the machine? What still needs a backend service? What should we know about putting AI into applications?”
AI blurs code ownership. “You have to think about consumed memory, at least right now,” she said. “It might be changing in the future, but at least at this point, it’s not possible to have general-purpose AI, which can do state of the art things, which you can do with ChatGPT.”
“I will start by saying that if you are a developer, you can start today,” Selvasingh said. “That’s very important. Every app will be reinvented with AI. There’ll be new apps built that were not possible before. The opportunities are super high.”
Figuring out new AI tools
Costlow asked, “When new tools appear, how do people figure out what they can use new AI tools for and what the particular problem domains are?”
Selvasingh noted that people can use Azure OpenAI services for generating and summarizing long-form content, building code and code snippets, improving search results, and more. At American Airline, he noted. they modernized their customer hub and implemented it on Azure with OpenAI. By using AI, they have reduced taxi times, saving fuel. They were able to save fuel and give passengers extra time to make their next flight.
They handle millions of real-time messages and service calls, now by using AI, they reduced the taxi times. They were able to save fuel and give passengers extra time to make their next flight…. AI is being used to create intelligent apps that drive amazing business outcomes, and it improves end user. If you are the passenger, if you are the driver, it’s exciting. If you’re a developer today, it’s an exciting time to be involved in AI development.
Asir Selvasingh
Libraries for starting in AI development
Costlow asked the panel to identify some of the key libraries that people can use and experiment with if they are just starting in AI development.
Schneider and Selvasigh were both impressed with LangChain4, which Schneider said seems current with most recent techniques like function calling that work their way into probably the more well-known LangChain Python library.
“LangChain4j is amazing,” Selvasigh said. “Microsoft also has Semantic Kernel. It can help you in building amazing apps faster.”
“If you’re a Java developer,” Schneider said, “don’t despair. “It’s not just for Python folks. There’s a lot of work that’s been done to catch up and, in some cases, surpass.”
If you’re a Java developer, don’t despair. It’s not just for Python folks. There’s a lot of work that’s been done to catch up and, in some cases, surpass.
Jonathan Schneider
AI integration into applications
Costlow asked how Java engineers can use AI directly for themselves, not necessarily integrated into the application for the user, but how Java developers can benefit themselves.
Schneider said organizations amass product elements until they have hundreds of millions of lines of code. They need a system that can make changes in a verifiably accurate way across all of it.
“At one banking customer, we did a single Java 8 to 17 migration just on one repo, and it affected 19,000 files,” Schneider added. “Imagine each one of those files having the potential for hallucination. We focus our energy on how we can use AI to help generate net-new recipes, instead of using AI to directly make the change. In other words, AI isn’t the cookie cutter itself. It’s the tool and die maker.”
______________________________________________________________________________
At one banking customer, we did a single Java 8 to 17 migration just on one repo, and it affected 19,000 files. Imagine each one of those files having the potential for hallucination.
______________________________________________________________________________
Improve DevOps productivity
Azul Intelligence Cloud provides actionable intelligence from production Java runtime data to efficiently prioritize vulnerable code for remediation and identify unused code for removal. It works with any JVM from any vendor or distribution including Azul, Oracle, Amazon, Microsoft, RedHat, and Temurin to slash time from unproductive tasks across an enterprise’s entire Java estate, resulting in a substantial improvement in DevOps productivity. Intelligence Cloud includes two services to help improve DevOps productivity.
Code Inventory® is the only solution that precisely catalogs what code runs in production across all of an enterprise’s Java workloads. It slashes the time and burden of maintaining and testing unused code, significantly improving developer productivity and ultimately saving money.
Azul Vulnerability Detection detects vulnerable code that is actually used versus simply present, eliminating false positives for efficient vulnerability triage. By detecting vulnerabilities at point of use in production, it fills a critical gap at the endpoint of the software supply chain and augments legacy tools for more accurate results.
Explore Intelligence Cloud
Improve DevOps productivity with Azul.
