Vintage SLAC accelerator software spreads its wings

Pioneering software called ACE3P was developed almost a quarter century ago to fine-tune the design of particle accelerators and their components. Now its latest incarnation is being adapted for scientific supercomputing and manufacturing design, thanks to partnerships between two companies and the Department of Energy’s SLAC National Accelerator Laboratory.

The collaborations are part of an Energy Department program called Small Business Innovation Research, or SBIR, that’s designed to be a win-win for both the lab and the community at large, said Matt Garrett, SLAC’s director of technology transfer and private partnerships.

“In these SBIR projects, technology developed by the labs and refined by our industrial partners goes out into the community for broad use, and then comes back to us to advance the facilities that are a crucial part of SLAC operations,” Garrett said.

By helping businesses advance their technologies and build markets, he added, the program also creates new domestic supply chains for things the lab —and in some cases, the wider community—needs.

ACE3P was developed at SLAC about two decades ago for making virtual prototypes of particle accelerator components that will work in real life, and it’s still widely used. ACE3P stands for Advanced Computational Electromagnetics 3D Parallel, reflecting the fact that it allows high-fidelity 3D simulations to run on thousands of computer processing units at once so researchers can solve large, complex problems faster.

In recent years, ACE3P has branched out to help researchers at universities and in industry perform simulations in other fields, including telecommunications and electromagnetic modeling of the human body, said Cho-Kuen Ng, a lead scientist at SLAC who helped develop ACE3P.

Today SLAC is working with two New York companies—Kitware and Simmetrix—to widen ACE3P’s reach. The goal is to make it much easier for researchers to use DOE supercomputers and to determine the ideal shape for accelerator components with design processes that can be applied to “just about everything,” says Simmetrix CEO Mark Beall—from airplane wings to cell phone batteries and injection molds for toys.

Supercomputing made simpler

SLAC’s work with Kitware goes back to 2015. The company creates open-source software platforms and customizes them for the needs of specific companies and government agencies; this last part is how it makes money on its freely available products.

In its current project with SLAC, the company is integrating one of its open-source platforms, Computational Model Builder, into the ACE3P software already in place at DOE’s National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory.

About 8,000 DOE-funded scientists use NERSC to perform unclassified research on a wide range of topics, including climate change, protein structure and the evolution of the universe. But as the size and complexity of those simulations grow, they’ve been getting harder and harder to manage.

Until recently, users had to type in codes—instructions for carrying out the simulations—by hand, while at the same time coordinating and keeping track of many intertwining threads of the project that each produce a tremendous volume of data, some of which has to be analyzed on the spot. Organizing and managing all this is becoming more and more of a hassle. And commercial interfaces that could help untangle the mess are not available for supercomputers, said John Tourtellott, Kitware’s principal investigator for the SLAC project.

Now that Computational Model Builder has been integrated into ACE3P, NERSC users can set the criteria for their simulations by filling out forms, pulling down menus and clicking instead of typing instructions. Then they can watch the simulation unfold and check the results before downloading the data to their own computer, Tourtellott said.

“Although we really can’t put a number on it, this has productivity benefits,” he said. “It can greatly reduce the amount of information that has to be entered by hand and the errors that happen as a result. It also leaves more time for the actual science.”

Kitware has also created a similar dashboard at DOE’s Los Alamos National Laboratory for researchers who use the lab’s Truchas software platform to simulate metal casting and 3D printing.

“The reason we started that project was not so much to save users time, but because we were encountering potential new users who would look at how much work their simulation would take and say, “It’s not worth my time’ and move on,” said Neil Carlson, a guest scientist at Los Alamos who led the Truchas project for eight years. “Creating the new interface is really a way to reduce that entry barrier.”

Another plus, Carlson said, is that the work Kitware did for the Los Alamos project was folded into Computational Model Builder so it’s available to everyone, “and that kind of floats everyone’s boat.”

The shape of things to come

What Kitware does for the supercomputer user experience, Simmetrix does for automatically generating meshes that represent geometric shapes in simulations.

Mechanical engineers use a mathematical technique called finite element analysis to see how the things they design—whether a small widget or an enormous accelerator part—will hold up under realistic operating temperatures, pressures, vibrations and so on. They can identify weak spots, change the shapes of components and repeat to come up with the optimal design in a computer before building a prototype. ACE3P has played a big role for decades in using these types of simulations to design accelerator components.

Finite element analysis breaks complex shapes into a bunch of much simpler ones, represented by meshes. The computer adds up the effects of each of these simple shapes on the performance of that particular design. Finer meshes allow more detailed simulations, but require a lot more computing time. Coarser meshes take less time but may not be as accurate. This mesh-generation process has to be repeated again and again to come up with an optimal design.

“If this was something you had to do manually, it would be incredibly tedious and a waste of time,” says Simmetrix CEO Beall. The only practical solution, he said, is to do it automatically.

SLAC researchers had developed a high-level process for predicting how to change a shape to produce a design that meets their requirements. But this process didn’t have a way to automatically predict which shape should be tested next or to automatically update the geometry and meshes for each new design. Simmetrix provided those missing parts to create a fully automatic process for updating and optimizing shapes and their meshes with ACE3P and similar design simulation platforms, Beall said. This will allow people to design better products faster and more cheaply, and it can be applied to virtually any product, including the manufacturing process itself.

Automating this function in ACE3P is big win for SLAC and for the company, which can build on whatever it creates for SLAC and market it to the public.

While the initial focus of the SLAC project is accelerator design for scientific facilities that may take decades to develop, Beall said, the model could also speed the design of accelerator technology for treating cancer and designing antennas and wireless devices.

“Both particle accelerators and medical devices use electromagnetic fields,” he said. “How efficient they are and how well they serve their purpose depends completely on the fields they make inside of them, which depends on the shape of the components.”

SLAC’s Ng said the SBIR project, which ended last year, has improved SLAC’s process for optimizing the shape of accelerator cavities with ACE3P, allowing designers to update the design parameters automatically rather than by trial and error. However, he said there is still some work to be done to make the process more broadly applicable for general use outside the lab.

Beall added that bits and pieces from the work that was done at SLAC have been integrated into Simmetrix products, including software the company has been selling for 25 years. “This project allowed us to develop new capabilities that will be very useful for our customers,” he said.