Home : Home : Warfare Centers : NSWC Dahlgren : Dahlgren Resources : Leading Edge : I&I Leading Edge : McHone
Mission Level Assessment and Evaluation:
Driving Fleet Exercises and Experimentation Based on Mission Effectiveness 
________________________________________________________

By Greg McHone

Mission Level Assessment and Evaluation (MLA&E) is a Mission Engineering approach to an iterative fleet/acquisition capabilities-based methodology, by which operational needs are defined through warfighter-validated mission architectures that influence joint and fleet rehearsals, exercises, and experiments. MLA&E’s warfighter integration intensive methodology evolves the Chief of Naval Operation’s (CNO’s) Integration and Interoperability (I&I) coordination with the fleet to one that is centered on warfighting capability and the technical-to-tactical excellence principle. Embedding MLA&E engineers and architects with the warfighter creates conditions for a philosophical change in evaluation, experimentation, and assessment that enables mission engineers to participate in the planning, execution, and analysis of fleet events to provide convincing evidence of warfighting capability through fleet-validated mission architectures. These validated operational-based architectures are then applied to existing programs and evolving technology to improve warfighting readiness. Under MLA&E, mission engineering starts with the desired warfighting capability, then scopes the platforms, systems, and performers necessary to achieve the desired warfighting capability.

Definition

The Vice Chief of Naval Operations (VCNO) established MLA&E activity under the I&I Charter, 19 DEC 2012. Scope, products, and resourcing of MLA&E were not yet defined in the charter, as compared to other prominent I&I activities, i.e., Warfare Capability Baseline (WCB) (weapon to target pairing kill-chain analysis) and Capability Solution Management (proposed solutions to gaps identified in killchain analysis). Since the early establishment of the Chief of Naval Operation’s I&I initiatives, MLA&E’s scope, approach, and deliverables have been vetted and refined through various fleet applications. The current version of the Integrated Capability Framework Operational Concept Document (ICF/OCD), dated 30 SEP 2013, defines MLA&E as understanding and quantifying how well a mission capability is currently achieved, and exploring, across the Doctrine, Organization, Training, Materiel, Leadership and Education, Personnel and Facilities (DOTMLPF) spectrum, the concepts for measurable mission improvement; enabling the insertion of I&I stakeholders into Fleet Exercises and Experimentation venues; and applying the ICF/OCD to capture data needed to evolve mission performance.

MLA&E consists of applying two related activities with different applications to fleet rehearsals and exercises. The “assessment” aspect of MLA&E identifies “baseline” warfighting capability through development of “as is” mission architectures derived from fielded systems. “Evaluation” is the application of MLA&E to fleet experiments to capture future “to be” mission architectures for non-materiel and materiel modifications. Through collaborative Fleet/ Engineering planning and execution processes, both assessment and evaluation applications of MLA&E have direct implications for near- to longterm holistic capability solutions that lead to improved warfighting readiness.

The MLA&E approach considers the larger I&I process, as promulgated in the VCNO’s I&I Charter, to embed the ICF/OCD data model application with fleet operations and realize the potential of MLA&E with Warfare Capability Baseline’s kill-chain assessments and Capability Solutions Management’s Integrated Capability Package development for proposed materiel and non-materiel solutions. MLA&E fleet-endorsed mission architectures represent the Deputy Assistant Secretary of the Navy (DASN) Research, Development, Testing and Evaluation (RDT&E) efforts to understand warfighter requirements for acquisition solutions that are available to the WCB, as well as address gaps identified in Warfighting Capability Assessments (WCA) that focus engineering and architectural resources for proposed integrated materiel and non-materiel solutions. Warfighting capability is represented by “Mission Success” in Figure 1.

The DASN RDT&E resourced the initial MLA&E effort where mission engineers from all Navy Systems Commands (SYSCOMs) were to validate the ICF/ OCD by applying the model to a major fleet exercise. The Fleet Experimentation Exercises and training category is one of eighteen uses defined by the ICF/ OCD. Once ICF/OCD was applied, MLA&E activity was to provide feedback on the data model’s utility for capturing fleet operations. To determine the best fleet operations candidate, MLA&E team reviewed a list of major exercises that provided the necessary mission area opportunities to apply ICF/OCD. Based upon concurrence from Commander, Pacific Fleet (PACFLT) Warfighting Assessment and Readiness (WAR), Valiant Shield 2012 was the major fleet venue selected for initial MLA&E application of the ICF/ OCD. The discussion that follows uses Valiant Shield 2012 as a representative use case.

Valiant Shield 2012

Valiant Shield is a recurring blue-red operational and fleet exercise leveraging joint participation of across service assets to counter a notional theater threat to a strategic ally in the Pacific theater of operations. The primary mission area focus of Valiant Shield 2012 revolved around Anti-Subsurface Warfare (ASW). However, from previous Valiant Shield exercises, several capability gaps in the Anti-Surface Warfare (ASuW) domain had been identified. Through coordination with PACFLT WAR staff, the MLA&E team began executing towards modeling ASuW operations. With PACFLT WAR, and given capability gaps identified during previous Valiant Shield exercises, the MLA&E team scoped their modeling effort down to the subject of Surface Action Group (SAG) operations.

Validating the ICF/OCD by applying it to a major fleet exercise was the principal I&I objective of MLA&E for Valiant Shield 2012. To accomplish this objective, the team examined the utility of the ICF data model to support fleet analysis, assessment, and requirements definition of mission performance. Additional objectives of MLA&E for Valiant Shield were to capture mission area architecture by identifying data attributes that establish system performance requirements and constraints, develop compliant architecture for capability integration, and establish fleet user interface views to graphically represent an applied ICF concept. The final objective was to support institutionalization of the CNO’s I&I mission assessment processes by establishing MLA&E application to fleet exercises and experiments in order to explore DOTMLPF solutions that meet near- and long-term operational challenges. Embedding I&I mission engineers with fleet users to identify and articulate operational needs for Research, Development, and Acquisition was necessary to capture MLA&E impact—Integrate fleet readiness with system development and performance.

Applying MLA&E during Valiant Shield 2012 provided a full spectrum of data collection activities in compliance with the ICF data model for architecture development. ICF specifies the following data elements and their relationships:

  • Authoritative warfighting capability data elements, relationships, and taxonomies
  • Canonical viewpoints and models to capture mission and system/platform capability data
  • Reference to authoritative guidance, standards, and sources for information
  • Configuration management guidance
  • Common standards for capturing and sharing framework
  • System/Mission Alignment Model (relationships, mappings)

In addition, the MLA&E activities exercised new roles and responsibilities of organizations that collaborate to construct the mission model and system/ platform configuration baselines for Valiant Shield. The MLA&E’s capabilities-based approaches proved that opposing forces capabilities are essential in determining warfighting capability and were therefore integrated into the Valiant Shield architecture via friendly force’s ability to defend and offensively engage. Opposing forces’ conceivable capabilities would desire to achieve similar capabilities to those of the U.S. Navy, at least at a high level of abstraction such that tasks from the Universal Navy Task List (UNTL) may apply to both opposing forces and friendly forces. Application of ICF to MLA&E by operational and acquisition stakeholders produced a foundation on which to build and evolve ME disciplines for improving fleet readiness. For example, Valiant Shield forces executed the sinking of a target ship (SINKEX) during 2012 exercise. Under traditional methodologies, the architecture would contain weapon, platform, targeting sensor, and target for the SINKEX. During this event, MLA&E produced architecture that identified the interdependencies of all participating platforms, systems, performers, and authoritative doctrine necessary to achieve the desired effect: sinking of the target ship. Figure 2 is a photograph of weapon impact on target ship.

Capturing fleet user perspective and determining mission area owner guiding principles were necessary to efficiently and effectively translate mission area requirements into acquisition and engineering processes. Under this principle, PACFLT WAR served as the mission area owner and provided governance over the operational data used for architectural development.

Mission Area Owner

Establishment of a “Mission Area Owner” is the keystone to MLA&E’s ability to obtain fleet validated and advocated mission threads that drive capability based acquisition:

"Fleet ownership of mission from definition,
through acquisition, to execution with ability to
assess and define mission needs to acquisition
community in order to ensure systems will meet
full mission needs or understand and agree to
limitations."
 

(PACFLT WAR: Valiant Shield 2012 Final Planning Conference)


Fundamental to capturing a fleet Mission Model in the ICF is the establishment of the guiding principles for mission area application. This ensures fleet and acquisition objectives and tasks stay true to agreed expectations by following clear and concise guiding principles to performing mission-level assessment processes for the application of the ICF data model to mission areas. The following are representative guiding principles of MLA&E agreed upon with PACFLT WAR in applying I&I activities to capture fleet user requirements:

  • Establish continuity of ICF/OCD across near, mid, and long-term to scope acquisition decisions and DOTMLPF solutions.
  • Provide convincing evidence to show operational benefit and impact.
  • Establish rigorous process to model operational environment that provides consistency and is responsive to change.
  • Ensure operational requirements are governed and managed by the fleet and understood by the acquisition community as a user.
  • Define interaction with fleet and acquisition stakeholders, consisting of periodic reviews to ensure needs and expectations are understood.
  • Allow mission architects to manage the fleet validated operational data and programs of record systems data for development of a common mission area-based architecture.

As the Mission Area Owner for Valiant Shield 2012 MLA&E, PACFLT WAR served as the Fleet Representative to the Mission Technical Baseline Authority. The Fleet Representative is the overall lead for Mission Technical Baseline development and validation, responsible for representing fleet requirements and interests, and to ensure the Mission Technical Baseline accurately reflects or adequately models operational reality (DOTMLPF). Mission Technical Baseline will be discussed in detail later in this article.

With a defined objective to generate a relationship between the fleet’s desired effects and the system functions that are necessary to achieve those effects, development of a capability-based model was required to produce mission area architectures.

Capabilities-Based Modeling Methodology

The MLA&E approach to applying the ICF/OCD to produce mission architectures is captured through a capabilities-based modeling methodology. Capabilities Based Modeling Methodology enables the operational, research, and acquisition communities to interpret and decompose the Commander’s guidance with consistency using the ICF data model that leads to efficient and effective mission architecture development. The MLA&E team established and followed a Capabilities Based Modeling Methodology to develop architectural products that model fleet operations starting at the fleet’s desired effects. Capabilities Based Modeling Methodology focuses on capabilities rather than individual point solutions or specific mission threads; the architecture can be used as a source of mission data and activity relationships for a multitude of different mission threads. Figure 3 graphically demonstrates the roadmap followed to create fleet validated mission threads in a reusable format for engineering and system design.

The objective of Capabilities Based Modeling Methodology is to generate a relationship between the Fleet’s desired effects and the system functions performed by material solutions generated by the acquisition community. Through this relationship, the fleet can clearly articulate the current state of deployed material solutions, their current assessment of system performance, and their current and future system needs as compared to particular desired effects and capabilities.

To maximize Fleet relationships and minimize impact to operational staff battle rhythm, Capabilities Based Modeling Methodology processes are integrated with existing fleet planning, assessment and analysis processes. Leveraging the efficiencies of existing fleet processes enables a more effective review of MLA&E products by operational planners. Returning to the Valiant Shield example, to ensure efficiencies optimize I&I impact, the MLA&E team integrated with exercise warfare and assessment syndicates to identify stakeholders, products and attributes. An additional tenant to this approach was to capture the existing PACFLT data collection process and compare it with I&I data, supplementing as needed to complete the architecture.

Establishing a common language between the fleet and mission engineers is essential for operational architecture (data) development. This operational/ acquisition taxonomy was realized through significant warfighter integration during Valiant Shield 2012. The data required to develop Department of Defense Architecture Framework (DODAF) Data Model 2 (DM2)-compliant views is comparable to the data needed by fleet readiness planners to perform analysis, complete assessments, and generate requirements. The variation between fleet and acquisition stakeholders is the way the data is organized and viewed based on unique user requirements. For example, Operational Views (OV), such as an OV-6a and OV-6c, would be viewed by fleet users as a “mission thread,” that is, a snapshot in time and uniquely tailored to their data requirements. Timelines may include initial conditions that impact how each activity is performed or how well the overall mission thread is able to satisfy the overall desired effect.

Desired effects are inherently linked to a capability. Under capability-based methodology, MLA&E began by identifying the purpose, objectives, scope, hypotheses, and desired effects of the Fleet and then derived from them the capabilities required to achieve mission success. Through a collaborative fleet/mission engineer effort, the AV-1 therefore evolved into Commander’s Guidance and Intent for the I&I activities during Valiant Shield. Collaborative AV-1 provided a common reference document to influence the schedule of events planning that supports achievement of overall assessment and acquisition objectives through the development of capabilities-based architecture. As such, the principal product from MLA&E efforts are mission-based architectures created using a capabilities based modeling that reflects overall warfighting readiness against a defined threat.

Mission Technical Baseline Architecture

Mission Technical Baseline (MTB) architecture development of mission threads has evolved to become the principal product for MLA&E related I&I activities. ICF MTB data elements and products capture operational requirements and identify required capabilities at the mission task and interface level that provides a top level Mission Area and Navy Tactical Tasks (NTA)-based task organization that provides a common operational framework for Navy I&I activities. This decomposition provides a mechanism to discover and define needed systems and platforms interfaces and behavior required to support end-to-end mission capability. MLA&E is the foundation for evolving mission engineering disciplines for improving warfighting readiness that provide the vehicle for developing mission architectures that support fleet and acquisition users. There are varying states and uses for an MTB that is determined by multiple stakeholders and their application of the data for specific products. Representative examples of MTB from MLA&E application include:

  • Snapshot of a specific instance assessment of deployed warfighting capability
  • Reference that baselines applicability to programs and Navy-wide deployed systems
  • Excursion derived from experimentation to influence research

Snapshot MTB of deployed mission architecture is developed to support fleet analysis, assessment, and requirements generation of existing warfighting capability. Mission architecture is derived from defined mission objectives and effects, and deployed Joint and Navy assets operating under theaterdefined doctrine to rehearse the ability to effectively execute operational and contingency plans. These mission effects are mapped up to authoritative mission capabilities, i.e., Joint Capability Areas (JCA), Required Operational Capabilities (ROC) and Projected Operational Environment (POE), Universal Naval Task List (UNTL), Navy Mission Essential Task List (NMETL), and down to platforms and systems that support achievement of those effects. Valiant Shield 2012 represents a specific instance of Mission Technical Baseline development that supports baseline assessment of deployed warfighting capability. MLA&E mission engineers provided mission threads developed from MTB to PACFLT WAR Data Analysis Working Group (DAWG) to provide convincing evidence of mission effectiveness for ASuW mission area. Theater-specific Mission Technical Baseline that supports assessment of warfighting capability is made available to the development and validation of “reference” architectures. Deployed and reference baseline MTBs are system agnostic as to how missions will be conducted.

Reference MTBs are “top-down” derived from authoritative mission capability requirements, i.e., JCA, ROC, UNTL, and NMETL. This differs from snapshot MTB in that a specific theater operational commander’s desired effects are the starting point and core of the mission thread. Reference MTB serves to align authoritative mission capability requirements with fielded systems and systems under development to better inform current acquisition processes and demonstrate applicability to improving warfighting readiness. Reference MTBs are consistent with approved concepts of employment used to provide more informed input to the Joint Capabilities Integrated Development System (JCIDS) and the Planning, Programming, Budgeting and Execution (PPB&E) processes. Offensive Anti-Surface Warfare (OASuW) Alternative of Analysis (AOA) used MTB derived from Valiant Shield snapshot mission threads to better understand fleet Long Range Anti-Ship Cruise Missile (ASCM) employment conditions that impact acquisition decisions.

Excursions from baseline and snapshot MTBs are made to use authoritative “as is” architectures to drive experimentation. Excursion MTB captures experimentation data for “to be,” or proposed warfighting capability to influence research, and technology and doctrine maturation. Excursion MTB MLA&E processes are embedded with Fleet Exercise and Experimentation working groups to refine mission level performance metrics and requirements for systems/ platforms that are measurable and testable within a mission capability framework before entering a formal acquisition program. In conjunction with Naval Warfare Doctrine Command (NWDC) and PACFLT WAR, MLA&E processes are applied to ongoing fleet exercises to gain further insights and to verify the methodology with the fleet in order to formalize ICF data model application to at-sea experimentation. The Reference MTB provides the operational conditions and desired warfighting capability to the Integrated Capability Technical Baseline (ICTB). The ICTB compares different architectures by allocating a set of systems (technical) that implement the reference architecture and execute the architecture to generate metrics that quantify the quality of the specific architecture. Sometimes an implementation of an MTB will modify workflows at both the operational activity sequence and functional activity sequence levels. Both snapshot and reference MTBs are useful for the development of an ICTB. Leveraging snapshot and reference mission architectures for Anti-Surface Warfare from Valiant Shield 2012, an ICTB was created to support the Littoral Combat Ship (LCS) Surface Module for Fast In-Shore Attack Craft (FIAC) mission that tied platform activities to fleet-validated mission thread during the Harry S. Truman Strike Group’s Sustainment Exercise.

Benefits of MTB expand to both fleet and SYSCOM organizations and work towards improving Navy acquisitions by producing better material solutions, and identifying when non-material solutions are factors to current or future capability gaps. Execution of the MTB allowed the MLA&E team to identify a coherent, fleet-endorsed approach to modeling operations that starts with capabilities and seeks to draw a connection between fleet desired effects and system performance. Participation in fleet events, as with Valiant Shield 2012, allowed the MLA&E’s mission engineering team to apply the ICF/OCD to an at-sea naval exercise and gain endorsement by Fleet advocates for the MLA&E process. The coordination between SYSCOMs and fleet organizations ensured that mission architecture products contain the appropriate scope and balance of activities necessary to accurately model end-to-end mission capabilities. Figure 4 depicts how system functions and required operational capabilities are related through an MTB created using a capability based model.

MLA&E is a DASN RDT&E initiative that is complementary to other I&I activities, while using a more operational perspective that is more closely tied to fleet activities and understanding of warfighter needs. The DASN RDT&E’s objective is to institutionalize and transition MLA&E processes to various fleet and acquisition stakeholders. Using Valiant Shield 2014 as the test bed, RDT&E is partnering with PACFLT WAR and other fleet readiness and experimentation commands, along with acquisition sponsors, Program Executive Offices, and SYSCOMs to assess MLA&E impact on overall warfighting capability. Participating organizations have unique objectives, from applying MLA&E processes to better inform and improve existing processes for improving warfighting readiness through a common data model for analysis, assessment, operational, and sponsored requirements generation, and system design considerations.

PACFLT WAR embedded the MLA&E engineers and architects with Valiant Shield Warfare Assessment Working Groups to assist in the generation of hypotheses, objectives, and assessment criteria to influence planning and execution of fleet events. The objective is to validate that Mission Technical Baseline architectures created during fleet assessment events will better inform analysis and requirements generation that leads to improved near-term warfighting readiness. Additionally, PACFLT WAR has endorsed MLA&E support for experimentation during Valiant Shield to determine how best to apply ICF/OCD data model to long-term assessment of warfighting capability.

The MLA&E team embedded with Valiant Shield Experimentation Working Group to leverage ASuW snapshot and reference Mission Technical Baselines for Long Range SAG Takedown excursion architecture development. This effort evolves MLA&E baseline “assessment” during Valiant Shield 2012 to “evaluation” through excursion architecture. MLA&E, an NWDC sponsored initiative, will serve as the model for linking multiple ASuW experiments during Valiant Shield. MLA&E initiative is the umbrella for separate Find, Fix, Finish, and Engage experiments for an end-to-end kill-chain experiment. NWDC is assessing the viability of MLA&E processes and ICF/ OCD data model for improving fleet experimentation.

Leveraging Valiant Shield 2014, a collaborative MLA&E and Naval Ordnance Safety and Security Activity (NOSSA) Weapon System Explosives Safety Review Board (WSESRB) team was established to identify safety and fratricide risks in the effective execution of fleet mission threads. This team, in support of DASN RDT&E will document a process that will integrate weapons safety engineering into Mission Engineering and document gaps in acquisition personnel, policy, and processes. Focus of MLA&E effort is on integrating Mission Engineering, Systems Engineering, and Software Systems Safety Engineering into the VCNO’s I&I Activity and system of system reviews utilizing the standing WSESRB processes. Integration effort will discover and document how best to review and assess the I&I characteristics of weaponized systems to understand safety risks, identify hazards and causal factors, assess mitigations, assess test and validation, and issue I&I Safety Findings or Actions to applicable programs under WSESRB review. Focus will be on fleet weapon systems, combat systems, and the respective network interfaces associated with fratricide. The overall objective is to apply MLA&E to further enable the institution of Mission-Level Engineering and I&I efforts in support of ASN RD&A’s strategic goals of fleet safety.

Commander, U.S. Fleet Forces N7 (Training) sponsored MLA&E support of Commander, Strike Forces Training Atlantic (CSFTL) assessors to identify improvements in data collection and analysis during fleet certification events, specifically, Composite Unit Training Exercises (COMPTUEX). Fleet Training and Readiness processes employ a NMETL mapped to specific Navy mission areas (capabilities) in a continuous improvement process called the Navy Warfare Training System. The owners of NMETLs use feedback from exercises, operations, and other events to improve how they articulate requirements, measure performance, certify readiness, and implement improvements. Effort to date has focused on FIAC defense during Harry S. Truman and George H.W. Bush Strike Groups’ Sustainment Exercises (SUSTEX) and COMPTUEX. It is expected that MLA&E application of the ICF/OCD data model will lead to better defined measures of effectiveness that lead to advanced levels of fleet readiness.

A key acquisition stakeholder from programs of record leveraging MLA&E’s Warfighter Integration is Program Executive Office for Integrated Weapon Systems (PEO-IWS). In conjunction with MLA&E, PEO-IWS is sponsoring Mission Technical Baseline development during Valiant Shield to produce an Integrated Capability Technical Baseline for Aegis Capability Baselines Surface Warfare mission area. Other acquisition stakeholders are Naval Sea Systems (NAVSEA) PEO-LCS for Surface and Mine Warfare Modules, Naval Air Systems (NAVAIR) Program Office for OASuW and Tomahawk Weapons Systems, thereby ensuring a broad understanding of operational needs by the acquisition community.

Conclusion

DASN RDT&E resources and MLA&E’s warfighter integration efforts to transition technical solutions to tactical capabilities create conditions for a philosophical change in evaluation, experimentation, and assessment that enable mission engineers to participate in the planning, execution, and analysis of fleet events to provide convincing evidence of warfighting capability through fleet-validated Mission Technical Baselines. These validated operational-based architectures are then applied to doctrine, existing programs, and evolving technology to improve warfighting readiness. Validation and transition of MLA&E processes and products with acquisition and fleet stakeholders institutionalizes Integration and Interoperability activities with existing operational readiness improvement processes.

Article Images



Figure 1. MLA&E Criteria



Figure 2. Sink Exercise (SINKEX)
                


Figure 3. Capabilities Based Roadmap



Figure 4. Capabilities Based Mission Model Transition