BUTLERVILLE, Ind. – Naval Surface Warfare Center, Crane Division (NSWC Crane) hosted the first Robust Artificial Intelligence Test Event (RAITE) at Muscatatuck Training Center from September 25 to September 28 with nearly 50 people in attendance.
RAITE is a cyber-physical, red and blue team Artificial Intelligence (AI) event that takes a unique approach to tackling the future of AI and how it directly impacts our armed forces and government. This approach is able to bring together industry, academia, and government experts to evaluate AI technology in a simulated real-world event.
Tyler Fitzsimmons, Machine Learning Subject Matter Expert for Platform Systems, and Dr. Tim Kelley, Scientist for Maneuver, Surveillance, & Engagement Systems, led this event. They believe providing real world experiences in a place such as Muscatatuck Training Center will better prepare our armed forces for real-life situations.
“RAITE is an important event because it helps us better understand the nature of continuous test and development of Machine Learning (ML) systems in realistic environments by allowing different teams to test the assumptions that a given system was developed under,” said Dr. Kelley. “Many weaknesses in AI/ML systems are not known until deployment because they are tested using the assumptions of the development team.”
For the event, NSWC Crane worked alongside multiple groups throughout the country. These groups included Defense Advanced Research Projects Agency (DARPA), Two Six, Indiana University (IU), University of Notre Dame, and Indiana University Purdue University Indianapolis (IUPUI).
Dr. David Crandall, professor of Computer Science and Director of the Luddy Artificial Intelligence Center, led a group of students from IU and believes that a hands on experience such as RAITE is incredibly important to the future of AI research.
“Students usually learn about the vulnerabilities of AI-based systems by reading research papers and textbooks, but it's difficult to really understand these issues unless you dive into the details yourself,” said Dr. Crandall. “This event was a really fun way of giving students hands-on experience with adversarial machine learning in a realistic and competitive – but still friendly and supportive – environment. The folks at Muscatatuck Training Center gave the competition a degree of realism and excitement that would be hard to replicate anywhere else.”
Alex Kiefer, a student of Dr. Crandall’s at IU, agrees that the live experience creates a unique opportunity to students learning and researching AI technology.
“RAITE was a unique opportunity to investigate the operational realities of deploying machine learning systems in production,” said Kiefer. “As Dr. Crandall mentioned, the majority of computer vision and AI research is conducted in a fairly controlled environment, and even so called "robust" systems are typically tested against only specific types of interference. This event introduced a number of complications to systems, some of which were expected, and others which highlighted inadequacies in the models which were not accounted for in their training. For students and researchers such as myself, this is a nice opportunity to expose ourselves and our systems to real-life conditions and better inform our research in robust, trustworthy systems.”
After groups arrived at Muscatatuck Training Center, they were divided into three different teams – red, blue, and purple. The red and blue teams ran the test while the purple teams monitored both groups to measure the results during the event.
To make the event seem as realistic as possible, a scenario was developed for the teams attending the event. The scenario described an impending attack on a company’s headquarters, both online and onsite. Blueprints were leaked online describing the technology used to build a series of undersea cables along with important details related to location, weaknesses, and correspondence.
Onsite, people were attempting to hack the security system to allow operatives into secure spaces undetected. It was the blue team’s responsibility to track the intruders and red team’s responsibility to intercept blue team.
Larry Gates, another student from Dr. Crandall’s group, found the event to be a great hands-on experience.
“I thought RAITE was very educational, especially the hands-on experience. Seeing the other attacks and hearing other’s descriptions of the models was informative and highlighted improvements for future models! The physical attacks really offered insight into computer vision weaknesses,” stated Gates.
Dr. Lauren Christopher is an associate professor of Electrical and Computer Engineering at IUPUI and temporary faculty for NSWC Crane.
Dr. Christopher led a group of students from IUPUI and they were tasked with gathering data from the experiment.
“The RAITE challenge brought together industry, DoD, and academia for a practical application of AI: person detection. We had the detectors running for both contested and uncontested environments. My academic students were able to gather data for AI anomaly detection, and consider how AI fails in contested environments,” explained Dr. Christopher. “It was a very valuable hands-on exercise that can lead to some papers in the near future.”
Artificial Intelligence work is continuing to grow at NSWC Crane, becoming more integrated into Electromagnetic Warfare, and events like RAITE can help increase the high-level technology that NSWC Crane delivers.
Fitzsimmons said, “AI will be a part of military technology, and providing opportunities to collaborate across academia, industry, and government is crucial. It is also critical to transition Trusted and Reliable AI systems that perform as expected, that our warfighter could consistently rely on.”
Overall, Fitzsimmons believes that RAITE was a success.
“The event went very well for a first iteration. Each team successfully completed their respective track algorithm implementation or red team attack. Different teams were able to get immediate feedback during RAITE and create new machine learning attacks and immediately deploy them in a real-time environment,” said Fitzsimmons.
Events like RAITE are increasing in importance due to the need of staying ahead of the competition. Within the next few years, Fitzsimmons would like to see this event expanded.
“We hope to grow RAITE to include other groups throughout the country and our NATO allies. By reaching out to more groups it allows those working in the AI space to have an open dialogue to discuss the problems that they are seeing because it is not going to be solved by any one group,” said Fitzsimmons.
About NSWC Crane | NSWC Crane is a naval laboratory and a field activity of Naval Sea Systems Command (NAVSEA) with mission areas in Expeditionary Warfare, Strategic Missions and Electromagnetic Warfare. The warfare center is responsible for multi-domain, multi- spectral, full life cycle support of technologies and systems enhancing capability to today's Warfighter.
Join Our Team! NAVSEA employs a diverse, highly trained, educated, and skilled workforce - from students and entry level employees to experienced professionals and individuals with disabilities. We support today's sophisticated Navy and Marine Corps ships, aircraft, weapon systems and computer systems. We are continuously looking for engineers, scientists, IT and cyber specialists, as well as trade and other support professionals to ensure the U.S. Navy can protect and defend America. Please contact NSWC Crane Human Resources at crane_recruiting@navy.mil.