Exploring Solutions Across Doctrine, Organization, Training, Materiel, Leadership and Education, Personnel and Facilities (DOTMLPF) Using Orchestrated Simulation through Modeling (OSM)
_________________________________________________________________________
By Sara Wallace and Mary Ann Cummings
Historically, the Naval Surface Warfare Center, Dahlgren Division (NSWCDD) has
solved many systems engineering challenges using deterministic, physics based Models
and Simulations (M&S) of the systems. For example, NSWCDD has developed high
fidelity M&S such as radar propagation models, six degrees of freedom models for
missile trajectories and blast fragmentation lethality models to support various
projects and sponsors. However, in order to perform analysis at the mission level,
it is necessary to model the multiple systems involved in achieving the mission
objective, especially the interactions between systems. New programs, such as the
Integration and Interoperability (I&I) efforts, require NSWCDD to begin to analyze
the interactions between the weapons, sensors, and weapon control systems.
Many of the current models used by the naval community analyze the interactions
by adding the other systems to their own system model, especially for
common systems such as threats, communications, etc. This introduces error
and redundancy by having each modeling entity create its own models, which are
neither shared nor reviewed between organizations. This dilemma can be solved
by having a government-owned, modular, and scalable
software framework capable of allowing the models
to interact in a common environment. NSWCDD has
invented and submitted a patent request for a software
framework called Orchestrated Simulation through
Modeling (OSM), which is truly a model agnostic and
capable of modeling the interactions between systems
by “plugging in” system models from government,
industry or academia. Plug-in is a term for a software
extension that adds capability or new features. Adding
system models via plug-ins allows software customization
for each project without making changes to the
framework. In addition, OSM enables a mission-level,
fast-running, qualitative assessment of the system’s
interactions. This assessment permits more focused
follow-on analytical deep dives with specialized tools
currently used for systems engineering. This article
describes the OSM framework; provides an example
of how OSM was used to support the Commander,
Operational Test and Evaluation Force (COMOPTEVFOR)
(Navy); and discusses the future of M&S to
support mission engineering.
Background—The Need for OSM and
Mission Level M&S
Across NSWCDD programs, M&S is typically
developed to support systems engineering for acquisition.
Currently, the system commands are typically
organized to develop and acquire systems (i.e., weapons,
sensors, weapon control systems, etc.) and platforms
(i.e., submarines, ships, etc.). Mission engineering
accounts for the interactions and performance of
the components, systems, and platforms that support
the mission(s) being performed. This relationship is
depicted in Figure 1.
As a rule, mission engineering studies aggregate
the performance of systems in order to perform studies
in a reasonable amount of time and effort. If the
detailed performance M&S for each platform, system
and component were used and each design parameter
analyzed, the time required to perform the study
would be astronomical and would not support timely
acquisition decisions. The approach detailed in this
article proposes use of behavioral models to represent
the systems. Behavioral models represent the systems
as independent entities that follow a defined state
diagram where the system moves from state to state
by stimulus from other systems or the environment.
This approach allows for emergent behavior, for example,
system performance, to be observed as systems
interact throughout a scenario. The behavior is not
predetermined or scripted. Each system’s behavior
can be unique for each run since it is dependent on
its interaction with other entities within the model.
One interaction may lead to another interaction and
so on. For example, a simple gun state machine could
have two states: fire and stand-by. To move between
the states, the gun could be stimulated by another
system, for example, a human, who has a state of
pulling the trigger. By using behavioral models, the
mission analysis can focus on the performance effects
of interactions of the systems.
Traditional modeling uses a set of equations to
dictate how the system operates and interacts, both
independently and with other systems. This equation-
based approach is used with many of the current
System of Systems (SoS) M&S tools used in the
Navy. SoS M&S tools often rely on the systems being
described by varying degrees of complexity. Further,
enormous energy is spent modeling system behaviors
down to the minutest detail and attempting to
fully exploit all possible outcomes of SoS behavior.
While the high fidelity M&S approach is quite feasible,
as stated earlier, it requires an abundance of time
and funding. This approach does not eliminate the
need for M&S at the lower levels since independent,
detailed characterization of system performance is
still required, especially when an issue is revealed at
the mission level. However, using behavioral models
allows program managers to be aware of a larger set
of solutions across the DOTMLPF spectrum, vice
the current paradigm of primarily looking at system
materiel solutions. This approach will be extremely
useful to explore “what-if” scenarios for mission engineering,
which is focused on how changes to tactics
and doctrine may close gaps in kill chains.
Analysis Definition—Integration &
Interoperability (I&I) Process
As discussed throughout this edition of Leading
Edge, the I&I process is different from most existing
acquisition processes in the Navy. First, the fleet
defines warfighting threads from operational plans
and/or needs. Second, COMOPTEVFOR and the warfare
centers characterize technical descriptions of the
warfighting thread. Third, the Integrated Capability
Framework (ICF) is used to identify possible solutions
to gaps. These mission thread assessments are performed
using M&S tools as well as operational test
data. Finally, the warfare analysis products (results)
become the basis for materiel and non-materiel solution
decision making in support of Office of the Chief
of Naval Operation’s (OPNAV’s) Capability Phasing
Plan (CPP).
OSM Overview
OSM is an evolutionary SoS framework that was
developed under NSWCDD in-house research funding
to investigate and solve computer science challenges
facing SoS modeling and mission engineering. OSM
follows the Discrete Event System (DEVS) specification
formalism which is a modular and hierarchical
formalism for modeling and analyzing timed event
systems. The OSM framework allows output visualizations
and DEVS M&S frames to be developed
separately as plug-ins and combined to form a mission
level model of many systems. OSM was designed to
use behavioral or agent-based models of the systems,
since this allows the analysts to interrogate the interactions
between the systems. There are five goals for
the OSM development effort as listed below.
- Create a government-owned, DEVS
based M&S framework using Object
Oriented Design (OOD), Agent Oriented
Programming (AOP), and plug-in computer
science design techniques to provide a
model agnostic, scalable tool that enables
integration and execution of multiple,
independently developed models that can
be used by government, industry, and
academia.
- Provide mission analysts with another (but
not the only) M&S tool in their tool-box
that supports M&S-based system analysis
and validation from the component level up
through the system, system-of-systems, and
mission-level.
- Minimize the effort required to integrate
independently developed models and
simulations.
- Facilitate the development of a rich set of
re-usable plug-ins.
- Facilitate the use and reuse of M&S
components to advance the discipline of
M&S based analysis.
OSM Framework Description
First and foremost, the OSM framework was
designed to be as simple as feasible. OSM provides
the minimum core set of services and functions that
will enable multiple plug-ins (models or simulations)
to exchange data and perform as an integrated program.
The OSM framework includes only the core
functionality and a common set of basic plug-ins that
users can select when creating a specific instantiation.
The OSM framework does not include user-specific
(system) plug-ins. Figure 2 depicts the framework
using several plug-in types.
The OSM framework is a core structure, or skeleton,
onto which users attach software and/or hardware
configuration items that are required to communicate
with each other. The core structure is a well-defined
set of classes, libraries, application program interfaces
(APIs), and development rules that provides a large
amount of reusable code and a systematic development
and integration approach. To investigate an
issue or solve analysis questions, users must formulate
their analytic approach and either build new or
re-use system models and output models on top of
the framework. Four main types of plug-ins are used
with the prototype OSM framework. They are the
system, experimental, simulator, and output plug-ins.
The first type of plug-in used in the OSM framework,
is the model plug-in. The model plug-ins
depicted in Figure 2 are: sensor, threat, ship, and
weapon. OSM uses behavior-based system models
called agents to observe and analyze the interactions
between the systems modeled. Prior to agent-based
modeling techniques, many system of systems simulations
relied upon systems dynamics modeling, which
is based on ordinary differential equations (ODEs) to
represent the interactions between systems. Using
agents allows the models to interact so the analysts can
explore the interrelationships and interdependencies
between the systems.
The second type of plug-in used is an experimental
plug-in, which follows the DEVS formalism. Experimental
plug-ins define pieces
of the scenario being modeled
that are not part of the system
behaviors. For example, experimental
plug-ins could include
weather and communications
models. While weather is not
part of a system model, changes
in weather will affect the system,
thus the interactions between
the weather plug-in and system
plug-in are important to model.
The third type of plug-in is
the simulator plug-in, which
also follows the DEVS formalism.
The kill chain analysis prototypes
have typically used a
discrete event plug-in, although
other simulator plug-in types,
including continuous and realtime
simulator plug-ins, have
been demonstrated in OSM.
The real-time simulation plug-in allows the user to
interact with the model to dictate its behavior. To
describe the difference between the simulator plug-ins,
an example could be a model which defines how a ship
travels from point A to point B. For the discrete event
plug-in, the ship transits by its defined behaviors in
the model. These behaviors could include making the
transit in the shortest distance, escorting another ship,
and/or following a pre-defined pathway of points. For
the real-time simulation, an analyst can override the
behavior by manual input of directions for the ship.
This shows potential to reduce model development
time to evaluate the effects of changes in behavior.
Instead of rewriting the model, the analyst can input
new directions for the ship while the simulation is
running.
The last type of plug-in is the output plug-in.
Output plug-ins are tools to aid in analysis. Many of
the output plug-ins are developed to visualize the simulation.
With so many systems interacting with each
other, visualization tools aid the analyst in identifying
issues. The prototype OSM development efforts have
used NASA’s Worldwind (a 3-D geospatial tool), and
OpenMap (2-D open source mapping tool) extensively
for visualization. Using externally developed, open
source tools allow analysts a plethora of benefits,
including the ability to leverage existing tools within
the open source community, to customize desired outputs,
and use other familiar software. Other common
output plug-ins include a metrics plug-in and a graph
plug-in. The metrics plug-in is populated by the analyst
with measures of performance, measures of effectiveness,
and/or other numerical measures that are
important to the specific analysis. The graph plug-in
displays these metrics in charts for easy viewing.
Since each analysis is unique, many output plug-ins
have been customized to support specific projects. In
addition, the OSM team developed an output plug-in
that extracts all data into a generic text file, allowing
the file to be imported and used to perform analysis
in external analysis tools.
Behavioral Models using Agents
This section provides a high level description of
agents. Agent-based modeling is important to the
OSM framework, since agents were used to create the
behavioral models for OSM. Agents, as used here, are
defined as autonomous decision-making entities with
diverse characteristics. In OSM, the rules for each
agent are defined through state diagrams and vary by
agent. The decision rules define models specific for the
agent of the external world, and carry a level of sophistication.
The rules employ memory of the agent’s
experiences, and are limited by the cognitive ability
or load of each agent. Each agent varies by its specific
attributes and available accumulated resources. This is
of particular significance because multiple, identically
coded agents can exist even if their behaviors during
the simulation are not identical.
The simulation is based on the interactions among
agents and each agent will have its own, unique set of
interaction experience. For example, for two, identically
coded, unmanned aerial vehicle (UAV) agents,
each may fly on a different flight path based on the
rule to hold a stand-off distance from other UAVs. In
another instance, the UAVs may have the same flight
path, but when launched at different times, the view
of the world will be different. Each UAV agent detects
moving threats at different points in time and makes
the decision to engage at a different time perhaps
based on being fired at or receiving a command to
fire. When faced with a decision, each agent makes an
independent decision that may or may not be the same
decision similar agents make. The decisions are based
on each agent’s interactions with the world and other
agents. The interactions can be simple or complex
depending on what questions the analyst is trying to
answer. Agents are beneficial to model interactions
since, unlike traditional equation-based models, there
is not a central authority or controller dictating how
the system operates, how the system is modeled, or
how the system moves from state to state in the state
diagrams. Using agents and behavioral models has
allowed interrogation into mission engineering challenges,
especially with system interactions that were
difficult to model with other M&S approaches.
OSM Instantiation
An instantiation of OSM is defined as a mission
scenario that is run to answer specific analysis questions
based on a defined analytical process for a given
mission thread. An OSM instantiation includes the
user-specific plug-ins (for systems, experimental, and
output) plus selected common and shared plug-ins
(output, experimental, and simulator) built on the
OSM framework. The plug-ins are selected based on
the analysis to be performed. During an instantiation,
the OSM framework provides the core services that
enable multiple user-provided plug-ins to run as an
integrated product. This is depicted in Figure 3.
An OSM instantiation, as depicted in Figure 3,
consists of five main parts. The technical description
of the problem to be assessed and the operational
warfighting need are used to form the analytical questions.
The analysis definition is created by the mission
thread along with a description of the results to be
collected. The OSM modeling framework allows the
plug-ins to execute together and is populated with a
combination of common, unique, and shared plug-ins.
Common plug-ins include output plug-ins, such as
visualization tools, which are useful to most applications
and generic enough to be suitable for a variety of
programs. A shared plug-in could be a missile model
created for one program, but the model is shared
with another program. A unique plug-in is a model
developed and used by only one program. Finally, the
analytical process is how the user chooses to execute
the instantiation. The analytical process may include
analysis assumptions, plans for technical reviews, and
other tasks deemed important by the user.
An example instantiation of OSM is a fleet exercise
planning and assessment model developed for
COMOPTEVFOR. COMOPTEVFOR’s vision was for
the model to be used before a test event for planning,
and after a test event for analyzing the event outcome.
Agents were created for naval systems playing a critical
role during the test event. This included guns (i.e., Phalanx
Close-In Weapons System (CIWS), 5-inch, etc.),
sensors, ship movement, helicopter movement, small
boat threats, and ship tactics. The agents were based
on the system behaviors including system interaction
and reaction to other systems. The system behaviors
were created by ship commanders and helicopter
pilots from COMOPTEVFOR and by NSWCDD’s
weapon experts and warfare analysts. The team documented
the system behaviors and relationships in
Systems Modeling Language, which was used to code
the system plug-ins in OSM. The behavioral models
also used data from high-fidelity, physics-based lethality
models, including probability of kill and timeof-
flight tables. Coupling high-fidelity algorithms
and look up tables to the OSM instantiation adds
the benefit of the high fidelity, lower system models,
without significant increases in run time. In addition,
COMOPTEVFOR wanted the interactions displayed
on a map and metrics captured. Both types of output
plug-ins were already created for other instantiations
of OSM and were available for reuse. Plug-ins that
can be shared across programs are called “common
services” plug-ins. Typically, these include methods
for displaying the data output from the model. The
COMOPTEVFOR instantiation of OSM is depicted
in Figure 4.
The OSM framework allowed COMOPTEVFOR
to build behavioral (agent-based) models of the
systems of interest and use common services plugins
from prior instantiations of OSM. In addition,
COMOPTEVFOR wanted a way to display output
metrics on the map plug-in. This included metrics
measuring which weapons were engaging a particular
threat, how many rounds had been fired, and which
weapon killed the threat.
Collaboration with NAVAIR Warfare Center
China Lake
During the creation of the OSM instantiation for
COMOPTEVFOR, the OSM team began a formal
collaboration with the Naval Air Systems Command
(NAVAIR) Warfare Center (NAWC) at China Lake
as a research effort between two systems commands
and warfare centers. The goal was to determine if an
outside organization could effectively use and develop
plug-ins, also called models, for OSM. The joint team
selected a common threat and kill chain with Navy
surface and air assets. NSWCDD built the surface
agents (ships, ship sensors, ship weapons, etc.) while
NAVAIR built the air agents (unmanned air vehicles,
air sensors, air weapons, etc.). NAVAIR was given the
OSM software specification and several examples
of prior OSM instantiations. The team held weekly
phone conferences, but, for the most part, NAVAIR
developed their behavioral models independently.
Since all the models were developed and complied
with the specification, each had a common interface.
At the end of the project, the two software development
teams met face to face and were able to bring
the models together in the OSM framework within
a matter of hours. After integration, the software
developers ran multiple scenarios to test how the
models reacted to various conditions to ensure the
integration was successful. A screen shot of the effort
is shown in Figure 5.
Figure 5 depicts the scenario containing models
from both NSWC Dahlgren and NAWC China Lake.
The round, colored icons represent various agents
depicting notional assets such as ships, unmanned
vehicles, and threats. The wedges represent notional
engagement areas for the weapons or coverage areas
for the sensors. This effort proved that software developers
external to the OSM were able to use the specification
and create behavioral models that would work
in the OSM framework with the models developed by
the OSM team. The effort also fostered collaboration
between the two warfare centers and provided the
OSM team with feedback on the performance of the
framework.
OSM to Support Future Mission Engineering
Projects
OSM enables the simulation of integrated
warfighting capabilities to assess multiple weapon/
target pairs and tactics. The current assessment is
a static assessment (on paper) of a single weapon,
single target pair. The future of mission engineering
will continue to include the assessment of gaps in
warfare ability (currently performed during the
Warfare Capability Baseline (WCB) studies), and
an assessment of possible solutions as part of the
Integrated Capability Plan (ICP). One concept for
increasing the capability of mission engineering is
to augment the WCB and ICP studies with cloud
charting, the Integrated Capability Framework
(ICF) and OSM. This futuristic, conceptual mission
engineering process is depicted in Figure 6.
The first step in the proposed I&I process is identical
to the existing I&I process. First, warfare gaps in
kill chains will be identified during the WCB studies.
Next, the kill chains assessed by the WCB will
be “brought to life” by developing system behavioral
models within the OSM framework by using
data from test events, models, as well as from system
architectures and other products
from the naval community
and the ICF. Technology
insertions, as identified in the
cloud charts, will be injected
into the OSM simulation to
determine the mission effect
of the technology insertion on
the kill chain. This capability
will be a valuable new tool,
allowing acquisition managers
to assess which technologies
produce the most significant
impact to the mission.
Further, the ICF of the future
will contain test data, system models, architectures
and acquisition data, all of which could be used to
answer a diverse set of analysis questions using OSM.
In addition, system and mission architectures could
be used as a basis for the behavioral-based models.
Architectures define the relationships and interactions
between systems. The future assessments and analysis
using OSM will allow for a more thorough assessment
of the design space for solution sets to support the ICP.
Another promising solution for future work is
for OSM to tie into other Department of Defense
(DoD) frameworks. OSM is designed for the analysis
of SoS using behavioral-based models. While
legacy models can be wrapped into OSM, many
high-fidelity, physics-based models were not designed
to work with other models. OSM has used look-up
tables from high-fidelity models in lieu of wrapping
the model, but other DoD frameworks may be able
to offer additional functionality by quickly wrapping
legacy, standalone models and hardware-in-the-loop
test articles. One DoD framework that shows promise
for linking legacy software is NAVAIR’s Architecture
Management Integration Environment (AMIE).
Funded by the Office of the Secretary of Defense
(OSD), AMIE is a proven framework to wrap existing
models and simulations of all types, including test
articles. The OSM team is currently collaborating with
the NAVAIR AMIE team. The OSM team proposes
to use AMIE to wrap legacy software into OSM and
is developing an OSM to AMIE interface. This would
allow other models and simulations to interact with
OSM through AMIE instead of being converted to
a behavioral-based model and wrapped into OSM.
Pairing multiple frameworks together may allow new
and legacy code to be integrated seamlessly into a
common simulation and achieve an ideal state for a
mission engineering analysis process.
Conclusions/Recommendations
Using behavioral models in addition to the typical
system performance models is advancing mission-
level assessment capability. Building the behavioral
models from the system architecture products
created by the programs of record is recommended.
Using architectures, such as DoD Architecture Framework
(DODAF) products, to define the behavioral
models ensures traceability and aids in validation
of the models. If a change is made in a system architecture
used in a behavioral model, the model will
also reflect this change. This is important for predicting
how the system behaviors will affect system
performance in the context of a mission. Of particular
interest would be to predict behavior patterns or
element/system states that lead to a degradation in
system performance. Once system behaviors can be
predicted, system M&S efforts can focus the use of
high-fidelity models to fully interrogate and solve the
system interoperability issues unveiled by the high
level model. The ability to characterize the system
behavior by predicting the interactions of the system
in order to predict system performance would allow
the acquisition community to optimize investment
strategies for future system development.
In conclusion, although the OSM framework is
still in the research stage, it shows great potential to
be used to support future mission engineering efforts.
The OSM framework is currently undergoing formal
software validation and design utilizing NSWCDD
internal investment funding. Once the validation is
complete, the framework can be freely distributed
within the Navy and across government agencies.
Using a common software framework to allow the
interaction of behavioral models created by separate
entities, while maintaining the intellectual property
of each model, will go a long way toward full collaboration
on future mission engineering programs.