Mission Level Assessment and Evaluation:
Driving Fleet Exercises and Experimentation Based on Mission Effectiveness
________________________________________________________
By Greg McHone
Mission Level Assessment and Evaluation (MLA&E) is a Mission Engineering approach
to an iterative fleet/acquisition capabilities-based methodology, by which operational
needs are defined through warfighter-validated mission architectures that influence
joint and fleet rehearsals, exercises, and experiments. MLA&E’s warfighter integration
intensive methodology evolves the Chief of Naval Operation’s (CNO’s)
Integration and Interoperability (I&I) coordination with the fleet to one that is
centered on warfighting capability and the technical-to-tactical excellence principle.
Embedding MLA&E engineers and architects with the warfighter creates
conditions for a philosophical change in evaluation, experimentation, and assessment
that enables mission engineers to participate in the planning, execution, and
analysis of fleet events to provide convincing evidence of warfighting capability
through fleet-validated mission architectures. These validated operational-based
architectures are then applied to existing programs and evolving technology to
improve warfighting readiness. Under MLA&E, mission engineering starts with
the desired warfighting capability, then scopes the platforms, systems, and performers
necessary to achieve the desired warfighting capability.
Definition
The Vice Chief of Naval Operations (VCNO)
established MLA&E activity under the I&I Charter,
19 DEC 2012. Scope, products, and resourcing of
MLA&E were not yet defined in the charter, as compared
to other prominent I&I activities, i.e., Warfare
Capability Baseline (WCB) (weapon to target pairing
kill-chain analysis) and Capability Solution Management
(proposed solutions to gaps identified in killchain
analysis). Since the early establishment of the
Chief of Naval Operation’s I&I initiatives, MLA&E’s
scope, approach, and deliverables have been vetted
and refined through various fleet applications. The
current version of the Integrated Capability Framework
Operational Concept Document (ICF/OCD),
dated 30 SEP 2013, defines MLA&E as understanding
and quantifying how well a mission capability is currently
achieved, and exploring, across the Doctrine,
Organization, Training, Materiel, Leadership and
Education, Personnel and Facilities (DOTMLPF)
spectrum, the concepts for measurable mission
improvement; enabling the insertion of I&I stakeholders
into Fleet Exercises and Experimentation
venues; and applying the ICF/OCD to capture data
needed to evolve mission performance.
MLA&E consists of applying two related activities
with different applications to fleet rehearsals
and exercises. The “assessment” aspect of MLA&E
identifies “baseline” warfighting capability through
development of “as is” mission architectures derived
from fielded systems. “Evaluation” is the application
of MLA&E to fleet experiments to capture future
“to be” mission architectures for non-materiel and
materiel modifications. Through collaborative Fleet/
Engineering planning and execution processes,
both assessment and evaluation applications of
MLA&E have direct implications for near- to longterm
holistic capability solutions that lead to improved
warfighting readiness.
The MLA&E approach considers the larger I&I
process, as promulgated in the VCNO’s I&I Charter, to
embed the ICF/OCD data model application with fleet
operations and realize the potential of MLA&E with
Warfare Capability Baseline’s kill-chain assessments
and Capability Solutions Management’s Integrated
Capability Package development for proposed materiel
and non-materiel solutions. MLA&E fleet-endorsed
mission architectures represent the Deputy Assistant
Secretary of the Navy (DASN) Research, Development,
Testing and Evaluation (RDT&E) efforts to
understand warfighter requirements for acquisition
solutions that are available to the WCB, as well as
address gaps identified in Warfighting Capability
Assessments (WCA) that focus engineering and architectural
resources for proposed integrated materiel
and non-materiel solutions. Warfighting capability
is represented by “Mission Success” in Figure 1.
The DASN RDT&E resourced the initial MLA&E
effort where mission engineers from all Navy Systems
Commands (SYSCOMs) were to validate the ICF/
OCD by applying the model to a major fleet exercise.
The Fleet Experimentation Exercises and training
category is one of eighteen uses defined by the ICF/
OCD. Once ICF/OCD was applied, MLA&E activity
was to provide feedback on the data model’s utility
for capturing fleet operations. To determine the best
fleet operations candidate, MLA&E team reviewed
a list of major exercises that provided the necessary
mission area opportunities to apply ICF/OCD. Based
upon concurrence from Commander, Pacific Fleet
(PACFLT) Warfighting Assessment and Readiness
(WAR), Valiant Shield 2012 was the major fleet venue
selected for initial MLA&E application of the ICF/
OCD. The discussion that follows uses Valiant Shield
2012 as a representative use case.
Valiant Shield 2012
Valiant Shield is a recurring blue-red operational
and fleet exercise leveraging joint participation of
across service assets to counter a notional theater
threat to a strategic ally in the Pacific theater of operations.
The primary mission area focus of Valiant
Shield 2012 revolved around Anti-Subsurface Warfare
(ASW). However, from previous Valiant Shield exercises,
several capability gaps in the Anti-Surface Warfare
(ASuW) domain had been identified. Through
coordination with PACFLT WAR staff, the MLA&E
team began executing towards modeling ASuW operations.
With PACFLT WAR, and given capability gaps
identified during previous Valiant Shield exercises, the
MLA&E team scoped their modeling effort down to
the subject of Surface Action Group (SAG) operations.
Validating the ICF/OCD by applying it to a
major fleet exercise was the principal I&I objective
of MLA&E for Valiant Shield 2012. To accomplish
this objective, the team examined the utility of the
ICF data model to support fleet analysis, assessment,
and requirements definition of mission performance.
Additional objectives of MLA&E for Valiant Shield
were to capture mission area architecture by identifying
data attributes that establish system performance
requirements and constraints, develop compliant
architecture for capability integration, and establish
fleet user interface views to graphically represent
an applied ICF concept. The final objective was to
support institutionalization of the CNO’s I&I mission
assessment processes by establishing MLA&E
application to fleet exercises and experiments in order
to explore DOTMLPF solutions that meet near- and
long-term operational challenges. Embedding I&I
mission engineers with fleet users to identify and articulate
operational needs for Research, Development,
and Acquisition was necessary to capture MLA&E
impact—Integrate fleet readiness with system development
and performance.
Applying MLA&E during Valiant Shield 2012 provided
a full spectrum of data collection activities in
compliance with the ICF data model for architecture
development. ICF specifies the following data elements
and their relationships:
- Authoritative warfighting capability data
elements, relationships, and taxonomies
- Canonical viewpoints and models to capture
mission and system/platform capability data
- Reference to authoritative guidance,
standards, and sources for information
- Configuration management guidance
- Common standards for capturing and
sharing framework
- System/Mission Alignment Model
(relationships, mappings)
In addition, the MLA&E activities exercised new
roles and responsibilities of organizations that collaborate
to construct the mission model and system/
platform configuration baselines for Valiant Shield.
The MLA&E’s capabilities-based approaches proved
that opposing forces capabilities are essential in determining
warfighting capability and were therefore
integrated into the Valiant Shield architecture via
friendly force’s ability to defend and offensively engage.
Opposing forces’ conceivable capabilities would desire
to achieve similar capabilities to those of the U.S. Navy,
at least at a high level of abstraction such that tasks
from the Universal Navy Task List (UNTL) may apply
to both opposing forces and friendly forces. Application
of ICF to MLA&E by operational and acquisition
stakeholders produced a foundation
on which to build and evolve
ME disciplines for improving fleet
readiness. For example, Valiant
Shield forces executed the sinking
of a target ship (SINKEX) during
2012 exercise. Under traditional
methodologies, the architecture
would contain weapon, platform,
targeting sensor, and target for
the SINKEX. During this event,
MLA&E produced architecture
that identified the interdependencies
of all participating platforms,
systems, performers, and
authoritative doctrine necessary
to achieve the desired effect: sinking
of the target ship. Figure 2 is a
photograph of weapon impact on
target ship.
Capturing fleet user perspective
and determining mission
area owner guiding principles
were necessary to efficiently and
effectively translate mission area requirements into
acquisition and engineering processes. Under this
principle, PACFLT WAR served as the mission area
owner and provided governance over the operational
data used for architectural development.
Mission Area Owner
Establishment of a “Mission Area Owner” is the
keystone to MLA&E’s ability to obtain fleet validated
and advocated mission threads that drive capability
based acquisition:
"Fleet ownership of mission from definition,
through acquisition, to execution with ability to
assess and define mission needs to acquisition
community in order to ensure systems will meet
full mission needs or understand and agree to
limitations."
(PACFLT WAR: Valiant Shield 2012 Final Planning Conference) |
Fundamental to capturing a fleet Mission Model in
the ICF is the establishment of the guiding principles
for mission area application. This ensures fleet and
acquisition objectives and tasks stay true to agreed
expectations by following clear and concise guiding
principles to performing mission-level assessment
processes for the application of the ICF data model to
mission areas. The following are representative guiding
principles of MLA&E agreed upon with PACFLT
WAR in applying I&I activities to capture fleet user
requirements:
- Establish continuity of ICF/OCD across near,
mid, and long-term to scope acquisition
decisions and DOTMLPF solutions.
- Provide convincing evidence to show
operational benefit and impact.
- Establish rigorous process to model
operational environment that provides
consistency and is responsive to change.
- Ensure operational requirements are
governed and managed by the fleet and
understood by the acquisition community as
a user.
- Define interaction with fleet and acquisition
stakeholders, consisting of periodic reviews
to ensure needs and expectations are
understood.
- Allow mission architects to manage the fleet
validated operational data and programs of
record systems data for development of a
common mission area-based architecture.
As the Mission Area Owner for Valiant Shield 2012
MLA&E, PACFLT WAR served as the Fleet Representative
to the Mission Technical Baseline Authority.
The Fleet Representative is the overall lead for Mission
Technical Baseline development and validation,
responsible for representing fleet requirements and
interests, and to ensure the Mission Technical Baseline
accurately reflects or adequately models operational
reality (DOTMLPF). Mission Technical Baseline will
be discussed in detail later in this article.
With a defined objective to generate a relationship
between the fleet’s desired effects and the system
functions that are necessary to achieve those effects,
development of a capability-based model was required
to produce mission area architectures.
Capabilities-Based Modeling Methodology
The MLA&E approach to applying the ICF/OCD
to produce mission architectures is captured through
a capabilities-based modeling methodology. Capabilities
Based Modeling Methodology enables the
operational, research, and acquisition communities to
interpret and decompose the Commander’s guidance
with consistency using the ICF data model that leads
to efficient and effective mission architecture development.
The MLA&E team established and followed a
Capabilities Based Modeling Methodology to develop
architectural products that model fleet operations
starting at the fleet’s desired effects. Capabilities Based
Modeling Methodology focuses on capabilities rather
than individual point solutions or specific mission
threads; the architecture can be used as a source of
mission data and activity relationships for a multitude
of different mission threads. Figure 3 graphically
demonstrates the roadmap followed to create fleet
validated mission threads in a reusable format for
engineering and system design.
The objective of Capabilities Based Modeling
Methodology is to generate a relationship between
the Fleet’s desired effects and the system functions
performed by material solutions generated by the
acquisition community. Through this relationship,
the fleet can clearly articulate the current state of
deployed material solutions, their current assessment
of system performance, and their current and future
system needs as compared to particular desired effects
and capabilities.
To maximize Fleet relationships and minimize
impact to operational staff battle rhythm, Capabilities
Based Modeling Methodology processes are integrated
with existing fleet planning, assessment and
analysis processes. Leveraging the efficiencies of existing
fleet processes enables a more effective review of
MLA&E products by operational planners. Returning
to the Valiant Shield example, to ensure efficiencies
optimize I&I impact, the MLA&E team integrated
with exercise warfare and assessment syndicates
to identify stakeholders, products and attributes. An
additional tenant to this approach was to capture
the existing PACFLT data collection process and compare
it with I&I data, supplementing as needed to
complete the architecture.
Establishing a common language between the fleet
and mission engineers is essential for operational
architecture (data) development. This operational/
acquisition taxonomy was realized through significant
warfighter integration during Valiant Shield 2012.
The data required to develop Department of Defense
Architecture Framework (DODAF) Data Model 2
(DM2)-compliant views is comparable to the data
needed by fleet readiness planners to perform analysis,
complete assessments, and generate requirements. The
variation between fleet and acquisition stakeholders
is the way the data is organized and viewed based on
unique user requirements. For example, Operational
Views (OV), such as an OV-6a and OV-6c, would be
viewed by fleet users as a “mission thread,” that is, a
snapshot in time and uniquely tailored to their data
requirements. Timelines may include initial conditions
that impact how each activity is performed or
how well the overall mission thread is able to satisfy
the overall desired effect.
Desired effects are inherently linked to a capability.
Under capability-based methodology, MLA&E
began by identifying the purpose, objectives, scope,
hypotheses, and desired effects of the Fleet and then
derived from them the capabilities required to achieve
mission success. Through a collaborative fleet/mission
engineer effort, the AV-1 therefore evolved into Commander’s
Guidance and Intent for the I&I activities
during Valiant Shield. Collaborative AV-1 provided a
common reference document to influence the schedule
of events planning that supports achievement of
overall assessment and acquisition objectives through
the development of capabilities-based architecture. As
such, the principal product from MLA&E efforts are
mission-based architectures created using a capabilities
based modeling that reflects overall warfighting
readiness against a defined threat.
Mission Technical Baseline Architecture
Mission Technical Baseline (MTB) architecture
development of mission threads has evolved to
become the principal product for MLA&E related
I&I activities. ICF MTB data elements and products
capture operational requirements and identify
required capabilities at the mission task and interface
level that provides a top level Mission Area and Navy
Tactical Tasks (NTA)-based task organization that
provides a common operational framework for Navy
I&I activities. This decomposition provides a mechanism
to discover and define needed systems and
platforms interfaces and behavior required to support
end-to-end mission capability. MLA&E is the foundation
for evolving mission engineering disciplines
for improving warfighting readiness that provide
the vehicle for developing mission architectures that
support fleet and acquisition users. There are varying
states and uses for an MTB that is determined by
multiple stakeholders and their application of the
data for specific products. Representative examples
of MTB from MLA&E application include:
- Snapshot of a specific instance assessment of
deployed warfighting capability
- Reference that baselines applicability to
programs and Navy-wide deployed systems
- Excursion derived from experimentation to
influence research
Snapshot MTB of deployed mission architecture
is developed to support fleet analysis, assessment,
and requirements generation of existing warfighting
capability. Mission architecture is derived from
defined mission objectives and effects, and deployed
Joint and Navy assets operating under theaterdefined
doctrine to rehearse the ability to effectively
execute operational and contingency plans. These
mission effects are mapped up to authoritative mission
capabilities, i.e., Joint Capability Areas (JCA),
Required Operational Capabilities (ROC) and Projected
Operational Environment (POE), Universal
Naval Task List (UNTL), Navy Mission Essential Task
List (NMETL), and down to platforms and systems
that support achievement of those effects. Valiant
Shield 2012 represents a specific instance of Mission
Technical Baseline development that supports
baseline assessment of deployed warfighting capability.
MLA&E mission engineers provided mission
threads developed from MTB to PACFLT WAR Data
Analysis Working Group (DAWG) to provide convincing
evidence of mission effectiveness for ASuW
mission area. Theater-specific Mission Technical Baseline
that supports assessment of warfighting capability
is made available to the development and validation
of “reference” architectures. Deployed and reference
baseline MTBs are system agnostic as to how missions
will be conducted.
Reference MTBs are “top-down” derived from
authoritative mission capability requirements, i.e.,
JCA, ROC, UNTL, and NMETL. This differs from
snapshot MTB in that a specific theater operational
commander’s desired effects are the starting point and
core of the mission thread. Reference MTB serves to
align authoritative mission capability requirements
with fielded systems and systems under development
to better inform current acquisition processes and
demonstrate applicability to improving warfighting
readiness. Reference MTBs are consistent with
approved concepts of employment used to provide
more informed input to the Joint Capabilities Integrated
Development System (JCIDS) and the Planning,
Programming, Budgeting and Execution (PPB&E)
processes. Offensive Anti-Surface Warfare (OASuW)
Alternative of Analysis (AOA) used MTB derived
from Valiant Shield snapshot mission threads to better
understand fleet Long Range Anti-Ship Cruise Missile
(ASCM) employment conditions that impact acquisition
decisions.
Excursions from baseline and snapshot MTBs are
made to use authoritative “as is” architectures to drive
experimentation. Excursion MTB captures experimentation
data for “to be,” or proposed warfighting
capability to influence research, and technology and
doctrine maturation. Excursion MTB MLA&E processes
are embedded with Fleet Exercise and Experimentation
working groups to refine mission level
performance metrics and requirements for systems/
platforms that are measurable and testable within
a mission capability framework before entering a
formal acquisition program. In conjunction with
Naval Warfare Doctrine Command (NWDC) and
PACFLT WAR, MLA&E processes are applied to ongoing
fleet exercises to gain further insights and to verify
the methodology with the fleet in order to formalize
ICF data model application to at-sea experimentation.
The Reference MTB provides the operational
conditions and desired warfighting capability to the
Integrated Capability Technical Baseline (ICTB). The
ICTB compares different architectures by allocating a
set of systems (technical) that implement the reference
architecture and execute the architecture to generate
metrics that quantify the quality of the specific
architecture. Sometimes an implementation of an
MTB will modify workflows at both the operational
activity sequence and functional activity sequence
levels. Both snapshot and reference MTBs are useful
for the development of an ICTB. Leveraging snapshot
and reference mission architectures for Anti-Surface
Warfare from Valiant Shield 2012, an ICTB was
created to support the Littoral Combat Ship (LCS)
Surface Module for Fast In-Shore Attack Craft (FIAC)
mission that tied platform activities to fleet-validated
mission thread during the Harry S. Truman Strike
Group’s Sustainment Exercise.
Benefits of MTB expand to both fleet and SYSCOM
organizations and work towards improving Navy
acquisitions by producing better material solutions,
and identifying when non-material solutions are
factors to current or future capability gaps. Execution
of the MTB allowed the MLA&E team to identify
a coherent, fleet-endorsed approach to modeling
operations that starts with capabilities and seeks to
draw a connection between fleet desired effects and
system performance. Participation in fleet events, as
with Valiant Shield 2012, allowed the MLA&E’s mission
engineering team to apply the ICF/OCD to an
at-sea naval exercise and gain endorsement by Fleet
advocates for the MLA&E process. The coordination
between SYSCOMs and fleet organizations ensured
that mission architecture products contain the appropriate
scope and balance of activities necessary to
accurately model end-to-end mission capabilities.
Figure 4 depicts how system functions and required
operational capabilities are related through an MTB
created using a capability based model.
MLA&E is a DASN RDT&E initiative that is complementary
to other I&I activities, while using a more
operational perspective that is more closely tied to
fleet activities and understanding of warfighter needs.
The DASN RDT&E’s objective is to institutionalize
and transition MLA&E processes to various fleet and
acquisition stakeholders. Using Valiant Shield 2014
as the test bed, RDT&E is partnering with PACFLT
WAR and other fleet readiness and experimentation
commands, along with acquisition sponsors,
Program Executive Offices, and SYSCOMs to assess
MLA&E impact on overall warfighting capability.
Participating organizations have unique objectives,
from applying MLA&E processes to better inform and
improve existing processes for improving warfighting
readiness through a common data model for analysis,
assessment, operational, and sponsored requirements
generation, and system design considerations.
PACFLT WAR embedded the MLA&E engineers
and architects with Valiant Shield Warfare Assessment
Working Groups to assist in the generation of
hypotheses, objectives, and assessment criteria to
influence planning and execution of fleet events. The
objective is to validate that Mission Technical Baseline
architectures created during fleet assessment events
will better inform analysis and requirements generation
that leads to improved near-term warfighting
readiness. Additionally, PACFLT WAR has endorsed
MLA&E support for experimentation during Valiant
Shield to determine how best to apply ICF/OCD
data model to long-term assessment of warfighting
capability.
The MLA&E team embedded with Valiant Shield
Experimentation Working Group to leverage ASuW
snapshot and reference Mission Technical Baselines
for Long Range SAG Takedown excursion architecture
development. This effort evolves MLA&E baseline
“assessment” during Valiant Shield 2012 to “evaluation”
through excursion architecture. MLA&E, an
NWDC sponsored initiative, will serve as the model
for linking multiple ASuW experiments during Valiant
Shield. MLA&E initiative is the umbrella for
separate Find, Fix, Finish, and Engage experiments
for an end-to-end kill-chain experiment. NWDC is
assessing the viability of MLA&E processes and ICF/
OCD data model for improving fleet experimentation.
Leveraging Valiant Shield 2014, a collaborative
MLA&E and Naval Ordnance Safety and Security
Activity (NOSSA) Weapon System Explosives Safety
Review Board (WSESRB) team was established to
identify safety and fratricide risks in the effective execution
of fleet mission threads. This team, in support
of DASN RDT&E will document a process that will
integrate weapons safety engineering into Mission
Engineering and document gaps in acquisition personnel,
policy, and processes. Focus of MLA&E effort
is on integrating Mission Engineering, Systems Engineering,
and Software Systems Safety Engineering into
the VCNO’s I&I Activity and system of system reviews
utilizing the standing WSESRB processes. Integration
effort will discover and document how best to review
and assess the I&I characteristics of weaponized systems
to understand safety risks, identify hazards and
causal factors, assess mitigations, assess test and validation,
and issue I&I Safety Findings or Actions to
applicable programs under WSESRB review. Focus
will be on fleet weapon systems, combat systems,
and the respective network interfaces associated with
fratricide. The overall objective is to apply MLA&E to
further enable the institution of Mission-Level Engineering
and I&I efforts in support of ASN RD&A’s
strategic goals of fleet safety.
Commander, U.S. Fleet Forces N7 (Training)
sponsored MLA&E support of Commander, Strike
Forces Training Atlantic (CSFTL) assessors to identify
improvements in data collection and analysis
during fleet certification events, specifically, Composite
Unit Training Exercises (COMPTUEX). Fleet
Training and Readiness processes employ a NMETL
mapped to specific Navy mission areas (capabilities)
in a continuous improvement process called the Navy
Warfare Training System. The owners of NMETLs
use feedback from exercises, operations, and other
events to improve how they articulate requirements,
measure performance, certify readiness, and implement
improvements. Effort to date has focused on
FIAC defense during Harry S. Truman and George
H.W. Bush Strike Groups’ Sustainment Exercises
(SUSTEX) and COMPTUEX. It is expected that
MLA&E application of the ICF/OCD data model
will lead to better defined measures of effectiveness
that lead to advanced levels of fleet readiness.
A key acquisition stakeholder from programs of
record leveraging MLA&E’s Warfighter Integration
is Program Executive Office for Integrated Weapon
Systems (PEO-IWS). In conjunction with MLA&E,
PEO-IWS is sponsoring Mission Technical Baseline
development during Valiant Shield to produce an
Integrated Capability Technical Baseline for Aegis
Capability Baselines Surface Warfare mission area.
Other acquisition stakeholders are Naval Sea Systems
(NAVSEA) PEO-LCS for Surface and Mine Warfare
Modules, Naval Air Systems (NAVAIR) Program
Office for OASuW and Tomahawk Weapons Systems,
thereby ensuring a broad understanding of operational
needs by the acquisition community.
Conclusion
DASN RDT&E resources and MLA&E’s warfighter
integration efforts to transition technical solutions to
tactical capabilities create conditions for a philosophical
change in evaluation, experimentation, and assessment
that enable mission engineers to participate in
the planning, execution, and analysis of fleet events to
provide convincing evidence of warfighting capability
through fleet-validated Mission Technical Baselines.
These validated operational-based architectures are
then applied to doctrine, existing programs, and
evolving technology to improve warfighting readiness.
Validation and transition of MLA&E processes and
products with acquisition and fleet stakeholders institutionalizes
Integration and Interoperability activities
with existing operational readiness improvement
processes.