Warfare Capability Baseline:
Assessing Operational Gaps
_______________________________________________________________
By David A. Clawson, Sara E. Wallace, Gregory D. Little, and Keith Wheeler
Mission engineering, including the desire to predict
and assess the warfighting capability of the Navy, is
nothing new. Analysts in the private and public sectors
have supported warfighting capability assessment
for years. Most studies have relied on subject-matter
experts and the use of modeling and simulation
(M&S) to assess system performance. While this
approach has proven to be invaluable to the acquisition
and engineering communities, experience has
shown it does not necessarily capture integration and
interoperability (I&I) issues that could impact the
performance of complex systems. Manually assessing
performance using technical expertise alone is not
feasible for most cases and, although M&S allows
analysts to model very complex system behavior,
models may be wrong, may not accurately represent
interfaces or important (possibly unknown) interactions
between systems, and may not accurately reflect
how the systems are used once deployed. Clearly, the
identification of gaps in our warfighting capabilities
requires that these and other issues to be considered
in the system assessment process. This article provides
an overview of the Warfare Capability Baseline
(WCB) assessment process, which identifies current
capability gaps using an approach primarily based
on operational test data.
Background
Investigation into a possible fleet issue necessitated
the assessment of current fleet capabilities grounded
with data collected from fleet exercises and system
operational and developmental tests. WCB began as a
pilot study led by Commander, Operational Test and
Evaluation Force (COMOPTEVFOR). Its mission is to
provide senior Navy decision makers with rigorous,
fact-based reports of current baseline systems-of-systems
warfighting capabilities “that are not available
anywhere else in the Navy.”1
Since the pilot study, the WCB assessment process
has been executed three times (referred to as “increments”).
Each successive increment has expanded
the scope of the assessment to include more threats
and more warfare areas. As of this writing, approximately
150 assessments have been performed across
six warfare areas: Ballistic Missile Defense (BMD), Air
Warfare (AW), Surface Warfare (SUW), Anti-Submarine
Warfare (ASW), Mine Warfare (MIW) and
Electronic Warfare (EW).
The WCB assessment process is derived from the
Naval Air Systems Command (NAVAIR)’s Horizontal
Integration and Capability Assessment Process
(HICAP). Some modification of the HICAP scoring
criteria was necessary to address the expanded scope
of the WCB assessment. Specifically, Threat Evaluation
and Weapon Assignment is a critical, complex activity
not explicitly identified in the HICAP functions.
Assessment Team
The WCB team comprises a diverse set of naval
agencies including COMOPTEVFOR; NAVAIR;
Naval Surface Warfare Center, Dahlgren Division
(NSWCDD); Naval Surface Warfare Center, Panama
City Division (NSWCPCD); Naval Undersea Warfare
Center (NUWC); Naval Air Weapons Center (NAWS)
China Lake; and the Space and Naval Warfare Command
(SPAWAR). An organizational chart depicting
the team hierarchy of the WCB assessment team is
displayed in Figure 1.
COMOPTEVFOR leads the overall WCB assessment.
The assessment team is divided into five groups:
surface; mine; undersea; air; and command, control,
and communication (C3), each led by a different warfare
center.
The lead at each warfare center is referred to as a
“Mission Engineer” (ME) and is responsible for working
with MEs from other warfare centers to develop
and review scoring criteria, provide the rationale
behind the assessment, and provide support for the
integrated assessment of all kill chains, including the
creation of the final report. Each ME also forms a
working group of “Principal Investigators” (PIs), who
are responsible for scoring systems in their particular
area of expertise. (e.g., the surface sensor PI will score
the performance of surface-based sensors such as
the SPY-1B). PIs also work with warfare center subject-
matter experts and keep program offices informed
as the assessment proceeds.
Each group is assigned a “Mission Integrator”
(MI), who is an active military COMOPTEVFOR
representative. MIs serve as liaisons between WCB
technical personnel and the warfighter community.
MIs’ roles include working with the fleet to develop
tactical situations (TACSITs) used in the assessment,
working with warfare centers to ensure current Navy
doctrine and tactics are represented, and supporting
the integrated assessment of kill chains.
Several other agencies played a role in the WCB.
These agencies include Program Executive Offices
that provided access to platform, sensor, weapon,
and network test data; and Commander U.S. Fleet
Forces Command (CFFC), who provided prioritized
weapon target pairs for assessment as well as the
mission context and criteria for each kill chain. The
Warfare Centers of Excellence also provided inputs
into developing TACSITs and provided feedback on
WCB findings.
Assessment Process
The WCB assessment process consists of several
steps:
- Defining and prioritizing weapon-target
pairs of interest
- Developing kill chains and TACSITs
- Developing scoring criteria
- Scoring kill chains
- Performing an integrated kill-chain
assessment
- Reporting assessment findings
A brief overview of each step is provided.
Defining and Prioritizing Weapon-Target Pairs
The WCB assessment process begins with a
fleet-prioritized list of weapon-target pairs (WTPs)
that identify a specific weapon for use against a specific
target. CFFC requests each fleet to provide a prioritized
list of WTPs that are of interest to them. The
CFFC and COMOPTEVFOR use these submissions
to determine the WTPs to examine in the next WCB
increment. For each WTP, the WCB team attempts
to answer the question: “Can this weapon be effectively
employed to achieve the desired effect against
this target under the circumstances described in
this TACSIT?”
Developing Kill Chains and TACSITs
Scored kill chains are the basic tool of WCB assessments,
are used to answer the above question, and
are the foundation for WCB products. A kill chain
consists of mission tasks or functions required to
successfully employ a specific weapon against a specific
threat and the platforms that could provide the
required functionality (e.g., target detection could be
done by an aircraft or a surface ship). In addition, the
kill chain includes the major decision nodes (e.g., the
decision to commit an aircraft to visually identify a
tracked object) as well as the communication links
required to transmit information between and within
units. The specific mission tasks and C3 nodes represented
in a kill chain are dependent on the mission
area being examined and are subject to change from
one increment to the next. In increment 3, the AW
kill chains consisted of eight mission tasks plus three
subtasks and six C3 nodes, as illustrated in Figure 2.
Given the number of combinations of mission
tasks and platforms and/or performers that could
carry out those tasks, there could be many “paths”
through a kill chain. Many of these paths will have
“broken” links, some will have “weak” but not broken
links, and (hopefully) some will be comprised of
all “strong” links. Since assessing all possible paths
through a kill chain is not generally possible, WCB
focuses on assessing the primary path, which represents
the Navy’s preferred path, based on current
doctrine, training, etc., and possibly one alternative
path that may circumvent weak or broken links in the
preferred path. In Figure 2, the primary path is indicated
by the solid black line connecting the scoring
nodes for Platform 1, and an alternative sensor path
is indicated by the dashed black lines connecting the
scoring nodes for Platform 2.
Kill chains are inherently tied to TACSITs, which
are vetted through an annual TACSIT development
effort with participation from fleet warfighters. These
TACSITs set the baseline year and provide warfighting
scenarios for kill chain assessments. The TACSIT contains
several important factors that must be considered
in the assessment, including the threat, the participating
platforms, the operational environments, and
expected methods of employment showing associated
geometries and timelines throughout mission execution.
In short, TACSITs provide the context within
which kill chains will be scored.
Developing Scoring Criteria
Each kill chain function must be scored in the
context of and constraints defined by its TACSIT.
Scoring criteria define what the systems must do. In
general, Green, Yellow, Red or White scores, as shown
in Table 1, are applied for each scored platform as
follows:
- Green: The platform provides the full level
of performance required by the TACSIT.
- Yellow: The platform provides a partial
or degraded level of performance in the
constraints defined by the TACSIT. For
instance, a Yellow score would be applied
to performance that occurred between a
desired and a minimum threshold, if both
were defined. In some circumstances, the
desired and minimum threshold are the
same, and a Yellow score does not exist,
i.e., performance is either Green or Red.
- Red: The platform fails to provide the
minimum level of performance required
by the TACSIT.
- White: Test data does not exist or is
insufficient to accurately score system
performance. The PI responsible for scoring
a given system or function makes the
decision as to whether there is sufficient
data to score system performance. In some
cases, accredited M&S is used to supplement
test results. If the M&S supports the trend
observed in the test data, then the function
will be scored; otherwise, the White score
remains in effect.
Critical
Measures |
Notional Scoring
Criteria |
Notional Score
and Rationale |
Overall
Task
Score |
| Launch_1 |
Is the correct weapon selected? |
GREEN: No issues observed in
testing |
YELLOW |
| Launch_2 |
Is all prelaunch data provided to
the weapon? |
YELLOW: Hardware
interface issues prevented launch
in x of y attempts |
| Launch_3 |
Is the launcher able to fire all
requested rounds for the
engagement? |
GREEN: No issues observed in
testing |
|
|
Table 1. Notional Critical Measures, Scores and Justifications for the Launch Subtask
|
|
Although, as explained above, the decision to
apply a White score is left to the discretion of the
responsible PI, the Green, Yellow and Red scores must
be justified and traceable to specific scoring criteria
developed for each mission area. For the AW mission
area, each mission task is broken down into a set
of critical measures. These critical measures are tied to
factors impacting mission success in a given TACSIT.
The ME and PIs develop the scoring criteria used to
score each of a task’s critical measures. Table 1 provides
a notional set of critical measures and scoring
criteria for the Launch subtask.
Scoring the Kill Chain
After receiving access to the test data, PIs must
sift through the data to determine which tests are
applicable to the mission task they are scoring, given
the conditions specified in the TACSIT. For example,
a PI may reject a test because the target drone is not
sufficiently representative of the threat. The PI then
uses the applicable test data and the appropriate scoring
criteria to score each of a task’s critical measures.
The lowest score assigned to any of a task’s critical
measures determines the task’s overall score for a
given platform. Notional scores and rationale are
provided in column 3 of Table 1.
Note: This data is for illustrative purposes only and
is not consistent with the actual WCB assessment. The
overall task scores (e.g., Green, Yellow, Red or White)
are applied to the scoring nodes and data links shown
in Figure 2 to provide a top-level summary of the kill
chain’s health.
It is important to understand that each mission task
in a kill chain is scored independently of the tasks that
precede it. For example, the Launch task will be scored
as if all tasks “upstream” were scored Green, so it is
possible to have a Green Launch task even though the
Track task is scored Red. This is done to avoid masking
the overall strength or weakness of a “downstream”
link, so that multiple issues affecting the successful
completion of a kill chain can be identified.
Performing an Integrated Kill Chain Assessment
In this step of the assessment process, all of the
individual platform and system scores and scoring
rationale are discussed in a working group consisting
of the entire WCB team. The national team critiques
the scores and justifications for every critical measure
in every kill chain to ensure they properly reflect the
scenario laid out in the TACSIT and accurately reflect
current capability in the TACSIT context. The weeklong
national team meeting allows the WCB team
to work through issues in real time and finalize all
scoring with the full consensus of the team across
all mission areas.
Reporting Assessment Findings
The primary purpose of a single scored kill chain is
to highlight broken and weak links or failed interoperability
and integration that would prevent success
in delivering a particular weapon against a particular
target with a specific mission employment in a given
environment. WCB findings are the delivered product
for highlighting such capability gaps in individual kill
chains. As the number of scored kill chains grows
in a particular mission area, including functional,
degraded, and broken kill chains, a broader picture
of warfighting capability becomes clear, and major
warfighting capability gaps identified.
In WCB Increments 1 and 2, a formal report documented
the major assumptions, scores, justifications
and findings for every assessed kill chain. In Increment
3, annotated electronic presentations served as
the final report. In all three increments, scoring and
justifications were placed into the WCB database
maintained by COMOPTEVFOR.
Interpreting Results
The current WCB assessment process is not perfect,
and care must be taken when interpreting the results.
In particular:
- Because overall platform scores are based
on a rollup of subsystem performance as
well as tactics, doctrine, etc., a score of RED
does not necessarily indicate the platform
vice the doctrine or tactic is at fault. PIs will
document the cause of the failure.
- Each AW kill chain to be assessed considers
only one weapon and one shooter. The
cumulative effect of multiple shooters or
additional weapons is not included in the
assessment. One implication of this decision
is that a “broken/failed” kill chain does not
necessarily imply that the targeted platform
is at risk.
- Similarly, since the kill chain is assessed
against a single-target raid, GREEN scores
do not necessarily indicate the kill chain will
perform well against a multi-target raid.
- Finally, in order to score a complete kill
chain, it is likely that data will be pieced
together from separate test events that
occurred under different conditions.
Consequently, the availability, applicability,
and consistency of test data must be
considered during the scoring process, when
interpreting the results, and when drawing
any conclusions.
Conclusion
The WCB process provides a valid method to assess
current warfare capability based on actual system performance
data from live-fire tests. The integration of
warfighters’ experience with analysts’ technical expertise
ensures that tactical needs will be addressed at
the technical level. Findings from previous increments
have established areas of focus for the acquisition and
science and technology communities that align with
the fleet’s highest priorities and concerns.