Top left banner Top right banner
Bottom left banner Bottom right banner
Home : Home : Warfare Centers : NSWC Dahlgren : Resources : Leading Edge : I&I Leading Edge : Wallace and Cummings
Exploring Solutions Across Doctrine, Organization, Training, Materiel, Leadership and Education, Personnel and Facilities (DOTMLPF) Using Orchestrated Simulation through Modeling (OSM)

By Sara Wallace and Mary Ann Cummings

Historically, the Naval Surface Warfare Center, Dahlgren Division (NSWCDD) has solved many systems engineering challenges using deterministic, physics based Models and Simulations (M&S) of the systems. For example, NSWCDD has developed high fidelity M&S such as radar propagation models, six degrees of freedom models for missile trajectories and blast fragmentation lethality models to support various projects and sponsors. However, in order to perform analysis at the mission level, it is necessary to model the multiple systems involved in achieving the mission objective, especially the interactions between systems. New programs, such as the Integration and Interoperability (I&I) efforts, require NSWCDD to begin to analyze the interactions between the weapons, sensors, and weapon control systems.

Many of the current models used by the naval community analyze the interactions by adding the other systems to their own system model, especially for common systems such as threats, communications, etc. This introduces error and redundancy by having each modeling entity create its own models, which are neither shared nor reviewed between organizations. This dilemma can be solved by having a government-owned, modular, and scalable software framework capable of allowing the models to interact in a common environment. NSWCDD has invented and submitted a patent request for a software framework called Orchestrated Simulation through Modeling (OSM), which is truly a model agnostic and capable of modeling the interactions between systems by “plugging in” system models from government, industry or academia. Plug-in is a term for a software extension that adds capability or new features. Adding system models via plug-ins allows software customization for each project without making changes to the framework. In addition, OSM enables a mission-level, fast-running, qualitative assessment of the system’s interactions. This assessment permits more focused follow-on analytical deep dives with specialized tools currently used for systems engineering. This article describes the OSM framework; provides an example of how OSM was used to support the Commander, Operational Test and Evaluation Force (COMOPTEVFOR) (Navy); and discusses the future of M&S to support mission engineering.

Background—The Need for OSM and Mission Level M&S

Across NSWCDD programs, M&S is typically developed to support systems engineering for acquisition. Currently, the system commands are typically organized to develop and acquire systems (i.e., weapons, sensors, weapon control systems, etc.) and platforms (i.e., submarines, ships, etc.). Mission engineering accounts for the interactions and performance of the components, systems, and platforms that support the mission(s) being performed. This relationship is depicted in Figure 1.

As a rule, mission engineering studies aggregate the performance of systems in order to perform studies in a reasonable amount of time and effort. If the detailed performance M&S for each platform, system and component were used and each design parameter analyzed, the time required to perform the study would be astronomical and would not support timely acquisition decisions. The approach detailed in this article proposes use of behavioral models to represent the systems. Behavioral models represent the systems as independent entities that follow a defined state diagram where the system moves from state to state by stimulus from other systems or the environment. This approach allows for emergent behavior, for example, system performance, to be observed as systems interact throughout a scenario. The behavior is not predetermined or scripted. Each system’s behavior can be unique for each run since it is dependent on its interaction with other entities within the model. One interaction may lead to another interaction and so on. For example, a simple gun state machine could have two states: fire and stand-by. To move between the states, the gun could be stimulated by another system, for example, a human, who has a state of pulling the trigger. By using behavioral models, the mission analysis can focus on the performance effects of interactions of the systems.

Traditional modeling uses a set of equations to dictate how the system operates and interacts, both independently and with other systems. This equation- based approach is used with many of the current System of Systems (SoS) M&S tools used in the Navy. SoS M&S tools often rely on the systems being described by varying degrees of complexity. Further, enormous energy is spent modeling system behaviors down to the minutest detail and attempting to fully exploit all possible outcomes of SoS behavior. While the high fidelity M&S approach is quite feasible, as stated earlier, it requires an abundance of time and funding. This approach does not eliminate the need for M&S at the lower levels since independent, detailed characterization of system performance is still required, especially when an issue is revealed at the mission level. However, using behavioral models allows program managers to be aware of a larger set of solutions across the DOTMLPF spectrum, vice the current paradigm of primarily looking at system materiel solutions. This approach will be extremely useful to explore “what-if” scenarios for mission engineering, which is focused on how changes to tactics and doctrine may close gaps in kill chains.

Analysis Definition—Integration & Interoperability (I&I) Process

As discussed throughout this edition of Leading Edge, the I&I process is different from most existing acquisition processes in the Navy. First, the fleet defines warfighting threads from operational plans and/or needs. Second, COMOPTEVFOR and the warfare centers characterize technical descriptions of the warfighting thread. Third, the Integrated Capability Framework (ICF) is used to identify possible solutions to gaps. These mission thread assessments are performed using M&S tools as well as operational test data. Finally, the warfare analysis products (results) become the basis for materiel and non-materiel solution decision making in support of Office of the Chief of Naval Operation’s (OPNAV’s) Capability Phasing Plan (CPP).

OSM Overview

OSM is an evolutionary SoS framework that was developed under NSWCDD in-house research funding to investigate and solve computer science challenges facing SoS modeling and mission engineering. OSM follows the Discrete Event System (DEVS) specification formalism which is a modular and hierarchical formalism for modeling and analyzing timed event systems. The OSM framework allows output visualizations and DEVS M&S frames to be developed separately as plug-ins and combined to form a mission level model of many systems. OSM was designed to use behavioral or agent-based models of the systems, since this allows the analysts to interrogate the interactions between the systems. There are five goals for the OSM development effort as listed below.

  1. Create a government-owned, DEVS based M&S framework using Object Oriented Design (OOD), Agent Oriented Programming (AOP), and plug-in computer science design techniques to provide a model agnostic, scalable tool that enables integration and execution of multiple, independently developed models that can be used by government, industry, and academia.
  2. Provide mission analysts with another (but not the only) M&S tool in their tool-box that supports M&S-based system analysis and validation from the component level up through the system, system-of-systems, and mission-level.
  3. Minimize the effort required to integrate independently developed models and simulations.
  4. Facilitate the development of a rich set of re-usable plug-ins.
  5. Facilitate the use and reuse of M&S components to advance the discipline of M&S based analysis.

OSM Framework Description

First and foremost, the OSM framework was designed to be as simple as feasible. OSM provides the minimum core set of services and functions that will enable multiple plug-ins (models or simulations) to exchange data and perform as an integrated program. The OSM framework includes only the core functionality and a common set of basic plug-ins that users can select when creating a specific instantiation. The OSM framework does not include user-specific (system) plug-ins. Figure 2 depicts the framework using several plug-in types.

The OSM framework is a core structure, or skeleton, onto which users attach software and/or hardware configuration items that are required to communicate with each other. The core structure is a well-defined set of classes, libraries, application program interfaces (APIs), and development rules that provides a large amount of reusable code and a systematic development and integration approach. To investigate an issue or solve analysis questions, users must formulate their analytic approach and either build new or re-use system models and output models on top of the framework. Four main types of plug-ins are used with the prototype OSM framework. They are the system, experimental, simulator, and output plug-ins.

The first type of plug-in used in the OSM framework, is the model plug-in. The model plug-ins depicted in Figure 2 are: sensor, threat, ship, and weapon. OSM uses behavior-based system models called agents to observe and analyze the interactions between the systems modeled. Prior to agent-based modeling techniques, many system of systems simulations relied upon systems dynamics modeling, which is based on ordinary differential equations (ODEs) to represent the interactions between systems. Using agents allows the models to interact so the analysts can explore the interrelationships and interdependencies between the systems.

The second type of plug-in used is an experimental plug-in, which follows the DEVS formalism. Experimental plug-ins define pieces of the scenario being modeled that are not part of the system behaviors. For example, experimental plug-ins could include weather and communications models. While weather is not part of a system model, changes in weather will affect the system, thus the interactions between the weather plug-in and system plug-in are important to model. The third type of plug-in is the simulator plug-in, which also follows the DEVS formalism. The kill chain analysis prototypes have typically used a discrete event plug-in, although other simulator plug-in types, including continuous and realtime simulator plug-ins, have been demonstrated in OSM. The real-time simulation plug-in allows the user to interact with the model to dictate its behavior. To describe the difference between the simulator plug-ins, an example could be a model which defines how a ship travels from point A to point B. For the discrete event plug-in, the ship transits by its defined behaviors in the model. These behaviors could include making the transit in the shortest distance, escorting another ship, and/or following a pre-defined pathway of points. For the real-time simulation, an analyst can override the behavior by manual input of directions for the ship. This shows potential to reduce model development time to evaluate the effects of changes in behavior. Instead of rewriting the model, the analyst can input new directions for the ship while the simulation is running.

The last type of plug-in is the output plug-in. Output plug-ins are tools to aid in analysis. Many of the output plug-ins are developed to visualize the simulation. With so many systems interacting with each other, visualization tools aid the analyst in identifying issues. The prototype OSM development efforts have used NASA’s Worldwind (a 3-D geospatial tool), and OpenMap (2-D open source mapping tool) extensively for visualization. Using externally developed, open source tools allow analysts a plethora of benefits, including the ability to leverage existing tools within the open source community, to customize desired outputs, and use other familiar software. Other common output plug-ins include a metrics plug-in and a graph plug-in. The metrics plug-in is populated by the analyst with measures of performance, measures of effectiveness, and/or other numerical measures that are important to the specific analysis. The graph plug-in displays these metrics in charts for easy viewing. Since each analysis is unique, many output plug-ins have been customized to support specific projects. In addition, the OSM team developed an output plug-in that extracts all data into a generic text file, allowing the file to be imported and used to perform analysis in external analysis tools.

Behavioral Models using Agents

This section provides a high level description of agents. Agent-based modeling is important to the OSM framework, since agents were used to create the behavioral models for OSM. Agents, as used here, are defined as autonomous decision-making entities with diverse characteristics. In OSM, the rules for each agent are defined through state diagrams and vary by agent. The decision rules define models specific for the agent of the external world, and carry a level of sophistication. The rules employ memory of the agent’s experiences, and are limited by the cognitive ability or load of each agent. Each agent varies by its specific attributes and available accumulated resources. This is of particular significance because multiple, identically coded agents can exist even if their behaviors during the simulation are not identical.

The simulation is based on the interactions among agents and each agent will have its own, unique set of interaction experience. For example, for two, identically coded, unmanned aerial vehicle (UAV) agents, each may fly on a different flight path based on the rule to hold a stand-off distance from other UAVs. In another instance, the UAVs may have the same flight path, but when launched at different times, the view of the world will be different. Each UAV agent detects moving threats at different points in time and makes the decision to engage at a different time perhaps based on being fired at or receiving a command to fire. When faced with a decision, each agent makes an independent decision that may or may not be the same decision similar agents make. The decisions are based on each agent’s interactions with the world and other agents. The interactions can be simple or complex depending on what questions the analyst is trying to answer. Agents are beneficial to model interactions since, unlike traditional equation-based models, there is not a central authority or controller dictating how the system operates, how the system is modeled, or how the system moves from state to state in the state diagrams. Using agents and behavioral models has allowed interrogation into mission engineering challenges, especially with system interactions that were difficult to model with other M&S approaches.

OSM Instantiation

An instantiation of OSM is defined as a mission scenario that is run to answer specific analysis questions based on a defined analytical process for a given mission thread. An OSM instantiation includes the user-specific plug-ins (for systems, experimental, and output) plus selected common and shared plug-ins (output, experimental, and simulator) built on the OSM framework. The plug-ins are selected based on the analysis to be performed. During an instantiation, the OSM framework provides the core services that enable multiple user-provided plug-ins to run as an integrated product. This is depicted in Figure 3.

An OSM instantiation, as depicted in Figure 3, consists of five main parts. The technical description of the problem to be assessed and the operational warfighting need are used to form the analytical questions. The analysis definition is created by the mission thread along with a description of the results to be collected. The OSM modeling framework allows the plug-ins to execute together and is populated with a combination of common, unique, and shared plug-ins. Common plug-ins include output plug-ins, such as visualization tools, which are useful to most applications and generic enough to be suitable for a variety of programs. A shared plug-in could be a missile model created for one program, but the model is shared with another program. A unique plug-in is a model developed and used by only one program. Finally, the analytical process is how the user chooses to execute the instantiation. The analytical process may include analysis assumptions, plans for technical reviews, and other tasks deemed important by the user.

An example instantiation of OSM is a fleet exercise planning and assessment model developed for COMOPTEVFOR. COMOPTEVFOR’s vision was for the model to be used before a test event for planning, and after a test event for analyzing the event outcome. Agents were created for naval systems playing a critical role during the test event. This included guns (i.e., Phalanx Close-In Weapons System (CIWS), 5-inch, etc.), sensors, ship movement, helicopter movement, small boat threats, and ship tactics. The agents were based on the system behaviors including system interaction and reaction to other systems. The system behaviors were created by ship commanders and helicopter pilots from COMOPTEVFOR and by NSWCDD’s weapon experts and warfare analysts. The team documented the system behaviors and relationships in Systems Modeling Language, which was used to code the system plug-ins in OSM. The behavioral models also used data from high-fidelity, physics-based lethality models, including probability of kill and timeof- flight tables. Coupling high-fidelity algorithms and look up tables to the OSM instantiation adds the benefit of the high fidelity, lower system models, without significant increases in run time. In addition, COMOPTEVFOR wanted the interactions displayed on a map and metrics captured. Both types of output plug-ins were already created for other instantiations of OSM and were available for reuse. Plug-ins that can be shared across programs are called “common services” plug-ins. Typically, these include methods for displaying the data output from the model. The COMOPTEVFOR instantiation of OSM is depicted in Figure 4.

The OSM framework allowed COMOPTEVFOR to build behavioral (agent-based) models of the systems of interest and use common services plugins from prior instantiations of OSM. In addition, COMOPTEVFOR wanted a way to display output metrics on the map plug-in. This included metrics measuring which weapons were engaging a particular threat, how many rounds had been fired, and which weapon killed the threat.

Collaboration with NAVAIR Warfare Center China Lake

During the creation of the OSM instantiation for COMOPTEVFOR, the OSM team began a formal collaboration with the Naval Air Systems Command (NAVAIR) Warfare Center (NAWC) at China Lake as a research effort between two systems commands and warfare centers. The goal was to determine if an outside organization could effectively use and develop plug-ins, also called models, for OSM. The joint team selected a common threat and kill chain with Navy surface and air assets. NSWCDD built the surface agents (ships, ship sensors, ship weapons, etc.) while NAVAIR built the air agents (unmanned air vehicles, air sensors, air weapons, etc.). NAVAIR was given the OSM software specification and several examples of prior OSM instantiations. The team held weekly phone conferences, but, for the most part, NAVAIR developed their behavioral models independently. Since all the models were developed and complied with the specification, each had a common interface. At the end of the project, the two software development teams met face to face and were able to bring the models together in the OSM framework within a matter of hours. After integration, the software developers ran multiple scenarios to test how the models reacted to various conditions to ensure the integration was successful. A screen shot of the effort is shown in Figure 5.

Figure 5 depicts the scenario containing models from both NSWC Dahlgren and NAWC China Lake. The round, colored icons represent various agents depicting notional assets such as ships, unmanned vehicles, and threats. The wedges represent notional engagement areas for the weapons or coverage areas for the sensors. This effort proved that software developers external to the OSM were able to use the specification and create behavioral models that would work in the OSM framework with the models developed by the OSM team. The effort also fostered collaboration between the two warfare centers and provided the OSM team with feedback on the performance of the framework.

OSM to Support Future Mission Engineering Projects

OSM enables the simulation of integrated warfighting capabilities to assess multiple weapon/ target pairs and tactics. The current assessment is a static assessment (on paper) of a single weapon, single target pair. The future of mission engineering will continue to include the assessment of gaps in warfare ability (currently performed during the Warfare Capability Baseline (WCB) studies), and an assessment of possible solutions as part of the Integrated Capability Plan (ICP). One concept for increasing the capability of mission engineering is to augment the WCB and ICP studies with cloud charting, the Integrated Capability Framework (ICF) and OSM. This futuristic, conceptual mission engineering process is depicted in Figure 6.

The first step in the proposed I&I process is identical to the existing I&I process. First, warfare gaps in kill chains will be identified during the WCB studies. Next, the kill chains assessed by the WCB will be “brought to life” by developing system behavioral models within the OSM framework by using data from test events, models, as well as from system architectures and other products from the naval community and the ICF. Technology insertions, as identified in the cloud charts, will be injected into the OSM simulation to determine the mission effect of the technology insertion on the kill chain. This capability will be a valuable new tool, allowing acquisition managers to assess which technologies produce the most significant impact to the mission. Further, the ICF of the future will contain test data, system models, architectures and acquisition data, all of which could be used to answer a diverse set of analysis questions using OSM. In addition, system and mission architectures could be used as a basis for the behavioral-based models. Architectures define the relationships and interactions between systems. The future assessments and analysis using OSM will allow for a more thorough assessment of the design space for solution sets to support the ICP.

Another promising solution for future work is for OSM to tie into other Department of Defense (DoD) frameworks. OSM is designed for the analysis of SoS using behavioral-based models. While legacy models can be wrapped into OSM, many high-fidelity, physics-based models were not designed to work with other models. OSM has used look-up tables from high-fidelity models in lieu of wrapping the model, but other DoD frameworks may be able to offer additional functionality by quickly wrapping legacy, standalone models and hardware-in-the-loop test articles. One DoD framework that shows promise for linking legacy software is NAVAIR’s Architecture Management Integration Environment (AMIE). Funded by the Office of the Secretary of Defense (OSD), AMIE is a proven framework to wrap existing models and simulations of all types, including test articles. The OSM team is currently collaborating with the NAVAIR AMIE team. The OSM team proposes to use AMIE to wrap legacy software into OSM and is developing an OSM to AMIE interface. This would allow other models and simulations to interact with OSM through AMIE instead of being converted to a behavioral-based model and wrapped into OSM. Pairing multiple frameworks together may allow new and legacy code to be integrated seamlessly into a common simulation and achieve an ideal state for a mission engineering analysis process.


Using behavioral models in addition to the typical system performance models is advancing mission- level assessment capability. Building the behavioral models from the system architecture products created by the programs of record is recommended. Using architectures, such as DoD Architecture Framework (DODAF) products, to define the behavioral models ensures traceability and aids in validation of the models. If a change is made in a system architecture used in a behavioral model, the model will also reflect this change. This is important for predicting how the system behaviors will affect system performance in the context of a mission. Of particular interest would be to predict behavior patterns or element/system states that lead to a degradation in system performance. Once system behaviors can be predicted, system M&S efforts can focus the use of high-fidelity models to fully interrogate and solve the system interoperability issues unveiled by the high level model. The ability to characterize the system behavior by predicting the interactions of the system in order to predict system performance would allow the acquisition community to optimize investment strategies for future system development.

In conclusion, although the OSM framework is still in the research stage, it shows great potential to be used to support future mission engineering efforts. The OSM framework is currently undergoing formal software validation and design utilizing NSWCDD internal investment funding. Once the validation is complete, the framework can be freely distributed within the Navy and across government agencies. Using a common software framework to allow the interaction of behavioral models created by separate entities, while maintaining the intellectual property of each model, will go a long way toward full collaboration on future mission engineering programs.

Article Images

Figure 1. The Systems Engineering Hierarchy

Figure 2. Depiction of Prototype OSM Framework with Plug-ins

Figure 3.
OSM Breakdown Structure

Figure 4. COMOPTEVFOR Instantiation of OSM

Figure 5. Screen Shot of OSM Collaboration with NAWC China Lake

Figure 6. OSM's Use in the Future I&I Process