I&I From a Cost Perspective
________________________________________________________
By Jonathan Brown and Sara Lloyd
In today’s budget climate, how can the Department
of Defense (DoD) ensure they are still developing and
procuring the equipment that our warfighters need?
By focusing on specific mission areas and looking
across the Doctrine, Organization, Training, Materiel,
Leadership and Education, Personnel and Facilities
(DOTMLPF) spectrum for solutions to address mission
capability gaps, Integration and Interoperability
(I&I) is uniquely suited to find the “knee in the
curve” and maximize the return on investment. A
key component to this optimization concept is the
ability to accurately predict costs across a spectrum
of mission solutions. An accurate prediction of costs
enables decision makers to efficiently utilize their
resources to attain the maximum performance for the
minimal cost, identify potential trade-offs, and identify
areas for cost reduction and/or avoidance. This
article discusses the role of cost estimating in the I&I
process, including the challenges of mission-level cost
estimates. It also provides an evaluation of current
cost estimating techniques that may be applicable to
mission-level analysis. Finally, it explores promising
areas of research that may mitigate some of the current
cost estimating challenges, such as the potential for
using existing I&I artifacts, technical metrics generated
during the I&I process, and Integrated Readiness
Level (IRL) as metrics for cost assessments. The integration
of cost analysis into the I&I process combines
research and development with operational support
and readiness in order to provide cutting edge cost
support to the warfighter.
Where is cost involved in I&I?
Figure 1 depicts the current I&I process. As previously
discussed in this edition of Leading Edge,
the Integrated Capability Package (ICP) prioritizes
the set of DOTMLPF solutions proposed to cover
the identified capability gap or set of gaps. Cost estimates
are required for trades studies and analysis of
alternatives as well as for each proposed solution in
the set. Having estimates of the required funding for
each solution allows decision makers to make more
informed decisions. As illustrated in Figure 2, tradeoffs
between cost, technical, and schedule are required
when faced with limited resources.
Challenges of I&I Cost Analysis
Cost Growth
While there are many different ways to measure
the extent and nature of cost growth, there are multiple
possible explanations and potential solutions for
improvement. Often, technology development and
acquisition programs are launched with too little focus
on system and mission-level requirements needed to
develop realistic cost estimates. Program managers
may have optimistic schedules and assumptions, such
as stable requirements, with corresponding low initial
cost estimates that are appealing to funding sources.
Realistically, however, significant requirements volatility
is common during the early acquisition phase,
which in turn causes significant cost growth. When
attention isn’t given to possible changes and growth in
requirements, budget overruns occur and programs
risk being canceled. In order to prevent programs
from being canceled, one must acknowledge and
plan for the likelihood that requirements will grow
and change. Canceled programs consume resources
with little or no return on investment; moreover,
they represent lost opportunities for viable programs
that may also fail due to lack of funding. The result
is a replicating program failure effect. The longer the
program continues before cancellation, the greater
the effect.
Requirements and Data
Squeaky wheels get the grease, and dedicated, tenacious
program managers are more likely to receive
program funding. Cost analysts rely on these system-
specific champions to help overcome obstacles to
prioritizing the cost team’s technical or data requests.
Costs cannot be determined for requirements that
are not defined in technical terms or do not indicate
the level of effort required. Therefore, cost analysts
depend upon obtaining data from the engineering and
technical teams to ascertain what costs will be affected
by the various alternatives. Most important to the
I&I efforts is to obtain candid technical assessments
of impact to other systems to determine the causal
effects on the cost. A strong program manager will set
priorities that ensure the engineering and technical
teams are thorough and responsive to the needs of
the cost analyst’s data requests.
Program Focused
Generally, cost estimates, similar to most other
systems engineering tasks, are program focused.
Cost estimating within the context of the I&I process
requires a paradigm shift from program focused
estimates to mission focused estimates in order to
accurately predict costs across a spectrum of mission
solutions. When implemented, this type of analysis is
similar to a large-scale Analysis of Alternatives and
requires the cost analyst to have breadth vs. depth
of analysis. Typically, cost analysis requires some
level of specialization. Successful cost analysts often
focus on one specific group of systems or platforms
or one area of cost analysis. Examples of specializations
include combat systems, ships, submarines,
rotary wing aircraft, etc. Often, the specific system
or platform is large; therefore the cost analyst gains
a breadth of experience in a multitude of subsystems
that exist within this area of specialization. Similar
to the I&I technical subject matter experts, I&I cost
analysts need to have not only a broad range of experience
but also a large network of experts to leverage
when required.
Subject Matter Experts
Expert opinion, or subject matter expert (SME)
input, is important to cost estimating by bringing a
wealth of experience and knowledge to the estimating
process. Many times, experts help to identify
analogous systems or provide data on labor hours or
testing requirements to build, operate, and maintain
a system. At the very least, an expert can provide his
or her opinion on cost drivers, functional form of a
regression and engineering rules-of-thumb on which
the cost analyst can consider in the estimate. When
working across multiple platforms, systems, and capabilities,
it is not possible to have a dedicated SME for
each potential solution on the team as compared to
a typical system acquisition cost IPT with dedicated
resources. A dedicated team with a robust network
of SMEs is needed to have access to the required
technical information, similar to a technical warrant
holder pyramid as depicted in Figure 3.
Opportunities for I&I Cost Analysis
Despite multiple challenges presented by integrating
cost analysis into the I&I process, many
opportunities exist. I&I related research will set the
conditions for providing more accurate cost estimates
early in the acquisition process (pre-Milestone and
Milestone A). Mission-level analysis reduces uncertainty,
provides better metrics and allows for a more
accurate cost model; hence, reducing the overall cost
and increasing the probability of program success.
Current cost estimating practices include providing
cost estimates to program management throughout
each phase of the acquisition life cycle. Cost estimates
are often required at the earliest stages of a program’s
development, i.e., Milestone A or earlier, when the
acquisition strategy and requirements are loosely
defined. This yields rough order of magnitude cost
studies, surrounded by a high degree of uncertainty
upon which important program decisions are made.
Furthermore, sometimes program-specific analysis
focuses on a single platform and a single system. This
may lead to a product based on a limited sight picture
rather than providing the customer with a holistic,
system-of-systems life-cycle analysis that is important
to providing the decision maker with various program
options (both technical and cost). This program-specific
analysis increases the risk that problems will
arise when the system or capability must interact
and interoperate with other systems to accomplish
its mission. Interoperability problems are becoming
a focus area and are expensive to fix once the system
or capability has been developed. An example of this
is the recent Accelerated Mid-Term Interoperability
Improvement Project (AMIIP) that was created to
address Fleet concerns about force-level interoperability;
specifically to address consistent correlation
between Cooperative Engagement Capability (CEC)
and Tactical Data Link (TDL) tracks.2 Mission-level
analysis helps to avoid these interoperability issues
and escalating costs by addressing system interactions
required to accomplish the mission early, at
the concept phase, in an acquisition program, thus
reducing potentially expensive interoperability “fixes”
in the future.
Another acquisition problem currently facing the
Navy is the development of technologies or capabilities
with no clear insertion point into the Fleet.
These “orphaned” programs have already sunk the
cost of development but provide no direct benefit
to the warfighter because they are never integrated
into a platform. More likely, they are installed onto a
platform but not integrated into the combat system.
The overall I&I process would reduce orphaned programs
by either identifying a specific mission gap
each capability will be addressing or, if no mission
need or no better solution is identified, the program
will not be recommended.
The final opportunity to consider relates to the
technical data available for cost estimation. The I&I
process puts an emphasis on upfront mission architecture.
This architecture process produces metrics
that have the potential to be used for cost analysis.
The emphasis on upfront mission architecture in the
I&I process ensures that the metrics are received early
often. This topic will be expanded on later in this
article, but some technical metrics that potentially
influence cost are information exchange messages
and Technology Readiness Levels (TRLs).
Evaluation of Current Cost Process
Suitability to I&I Cost Analysis
The following section summarizes the current cost
analysis process and identifies the suitability to its
applicability to the I&I process. Figure 4 represents the
Government Accountability Office’s (GAO) cost estimating
process. Often, cost analysts apply a tailored
version of the GAO’s Cost Estimating Process that is
most applicable to the program and associated tasks.
Define the Estimate and Develop the Estimating Plan
|
|
Estimating by
ANALOGY
|
Estimating by
PARAMETRICS
|
Estimating by
ENGINEERING
|
Estimating by
ACTUAL COSTS
|
| What is it? |
Single value from single data point |
Measure of trends across programs |
Detailed build-up of Lab, Mtl & OH $ |
An extrapolation of current program cost |
| When used? |
Early in Program Pre-Milestone A &
Milestone A |
Milestone B |
Late in Program Milestone C & LRIP |
LRIP
Full Rate Production |
| How is it done? |
Adjust analogous system cost or create cost factor |
Apply statistical methods to cost of 2 or more systems (i.e., Develop a CER) |
Estimate at lowest cost level & sum costs by WBS |
Use trend from your current contract to estimate your final system $ |
| PROS |
Fast, inexpensive
Easy to change |
Based on >1 data point=>less risky
Can measure error
Easy for what-ifs |
More details enables better visibility into cost drivers
|
Most costs are known
CAIG prefers over other methods |
|
CONS
|
Based on single historical data point
+>risky!
Trends to be more subjective |
Constrained by amount & quality of data (GIGO)
Statistics can be misleading |
Labor Intensive
Slow, Expensive
Can lose sight of "big picture"
|
Usually too late to use actual costs to adjust or build budget
Not a 1:1 correlation of prototype-to-production costs |
|
|
Table 1. Cost Estimating Methodologies4
|
One of the most critical first steps is to define the
scope of the cost estimating effort and to set the due
date for the completed cost estimate. Upfront planning
will still be required for any cost estimate, I&I
or otherwise.
Define the Program
I&I cost estimates will require the analyst to define
the program. This will be more difficult for an I&I
program because instead of a single program, system
or capability, the I&I solutions will likely be a range
of options to address the given mission. For traditional
acquisition programs, this defining information
comes from a Cost Analysis Requirements Description
(CARD) or other written documentation from
the program office. When the CARD is unavailable
or does not include enough information, the cost
analyst and technical team must work together to
identify constraints and reasonable assumptions that
will define the effort. For I&I, similar information is
still required; however, there will no longer be a single
source document or program office. Furthermore, it
is unlikely that the depth of information required in a
CARD will be available for each option. Similar to, but
broader than an AoA, a prioritized list of information
required for an I&I cost estimate must be provided
by a network of program offices and SMEs. The list
of information required could be far more extensive
than other early estimation efforts.
Determine the Estimating Structure
An I&I cost estimate will still require the development
of a Work Breakdown Structure and identification
of an estimating method for each. This part of the
process will likely be similar to other cost estimates.
Identify Ground Rules & Assumptions (GR&A)
While the actual ground Rules and Assumptions
(GR&A) developed for an I&I cost estimate may vary
from a traditional cost estimate, the process to identify
them will be similar.
Obtain The Data
Data gathering is the most difficult task in the cost
estimating process. Data collected falls into two basic
categories: data from previous programs used as a
basis for cost relationships and technical data for the
current program used to project future costs from the
historical actuals. For a typical cost estimate, the analyst
must identify SMEs, analogous systems, previous
cost estimates on the topic, relevant databases, Cost
Analysis Requirements Description, and pertinent
program and engineering data from the program
office. As mentioned above, for I&I cost estimates, this
process is difficult since there are multiple platforms
and systems involved to address a specific mission.
However, there are potential new sources of technical
data available from the I&I mission architecture to
help address this complexity.
Develop the Point Estimate
There are four accepted methods by which
cost estimates are generated: analogy, parametric,
engineering, and actuals. Table 1 summarizes the
four methods.
Analogy
Analogy entails using a single comparison value
from a system with similar characteristics. This
method is most often used early in the acquisition
phase when there is little or no data available on the
new system. While it is relatively quick and easy to
use compared to with other methods, the results may
be generalized and subjective.
Parametrics
The parametric approach uses statistical means
to measure trends across multiple programs. This
method requires a large amount of data; however,
the outcome yields measurable and tractable results.
Engineering Estimate
An engineering estimate is a detailed, bottom-up
approach which is usually used later in the acquisition
process. Gathering the data required can be a slow
process that is labor intensive but costs are estimated
at a high level of detail which will provide better
insight into cost drivers.
Actual Costs
Actual costs or “actuals” are an extrapolation from
the current data of the costs that have been incurred.
These costs are usually the best and most precise costs;
however, often the data becomes available too late in
the acquisition cycle to be useful.
While the actual analogies or technical metrics
used in a parametric estimate may differ from a traditional
cost estimate to an I&I cost estimate, there are
no expected variances to the four acceptable methods.
Given the information required to use an engineering
buildup or actual cost, employing these methods
will not likely be frequently used in the I&I process.
The fact that the I&I analysis is conducted early in
the acquisition life cycle will necessitate a larger use
of analogies and parametrics. However, this is not
absolute. For example, if an existing technology is
proposed as a solution in an ICP, the actual costs of
that item would then be used in the I&I cost estimate.
Final Steps in the Estimating Process
The final steps, shown in Figure 4, of uncertainty
analysis, presentation to management, and updating
the estimate will require no anticipated changes from
the traditional process. However, there is a subtlety
to be emphasized. A typical cost estimate for a program
or system usually resides under a single System
Command (SYSCOM) and program office. There may
be additional stakeholders, but the SYSCOM and
program office are responsible for providing the final
review of the cost (SYSCOM Cost Organization) and
technical (program office) assumptions and methodologies.
By contrast, an I&I cost estimate could
include multiple programs or systems that potentially
cross multiple SYSCOMs (NAVSEA, NAVAIR,
MARCORSYSCOM, etc.). Who would provide the
final review and be responsible for the validity of an
estimate that crosses multiple SYSCOMs and program
offices? Some options for answering this question
could include designating a lead SYSCOM for each
ICP. The ICP lead SYSCOM would be responsible for
approving the entire package. An alternate approach
would be to require individual SYSCOM sign off on
the respective technical area of a single ICP. The final
process is yet to be determined, but responsibility
of the I&I cost and technical review will need to be
established.
Cutting Edge Research and Promising Topics
Cost estimates for I&I products present several
challenges to the cost estimator. These challenges also
open up new cost research areas to help mitigate or
overcome these challenges. The following topic areas
are of particular interest.
Messages, Software Estimating, And Analysis
Modeling (SEAM)
A 2009 study conducted at NSWCDD, Software
Estimating and Analysis Modeling (SEAM), investigated
current software size estimating processes
involving Software Lines of Code (SLOC), recommended
improvements, and suggested alternatives.
One promising alternative technical metric suggested
was the measurement of messages. Messages are the
packets of information sent from one software component
to another. In the research, a preliminary
statistical relationship was established between the
number of messages and the number of SLOC. While
this technical metric for cost measurement requires
more research before utilized for cost analysis, it is
very promising from an I&I perspective. SLOC is the
primary software sizing metric for cost analysis, but
as mentioned above, is not likely to be available early
on during ICP development. Because of the emphasis
on mission architecture, message counts are likely to
be available for I&I analysis. If the preliminary results
remain stable under further research, this new metric
will provide a link between I&I mission architectures,
SLOC and cost. To capitalize on this research would
require the I&I ICP development process to include
sufficient time to conduct the architecture development
at a low enough level to be useful to the cost
estimate.
Readiness Levels
Previous GAO assessments have correlated low
technology maturity with programmatic problems;
programs that began development with immature
technologies saw a 32% cost growth and 20 month
schedule growth.5 There is a growing demand for
cost analysis during the early phases of the acquisition
cycle when the level of technology maturity is
low. Therefore, a promising area of future research,
and one that serves to provide significant dividends
in the future, includes relating cost to TRL, or any
of the various readiness levels, in the face of limited
data (see Figure 5).
“Maturing new technology before it is included on
a product is perhaps the most important determinant
of the success of the eventual product—or weapon
system.” The GAO (1999) also encouraged the use
of “a disciplined and knowledge based approach of
assessing technology maturity, such as TRLs, DoDwide
(p. 7).”6
Types of Readiness Levels
There are multiple types of readiness levels that
may be relatable to cost. Technology Readiness Level
(TRL) is probably the most commonly known readiness
level because it is used as the DoD standard
evaluation system to categorize hardware, software
or system concepts. TRL is simply a measure of an individual technology
and it does not account for readiness for use
in the system context. The definition of TRL varies
across the Department of Defense, Department of
Homeland Security, Department of Energy, National
Aeronautical Space Administration (NASA), and
European Space Agency. Integration Readiness Level
(IRL) is the status of the connections between the
technologies. System Readiness Level (SRL) is the
system-level appraisal of development maturity and
is a function of IRL and SRL. Manufacturing readiness
levels (MRLs) are quantitative measures used to
assess the maturity of a given technology, component
or system from a manufacturing perspective. Finally
Cost Readiness Levels (CRLs), as used by NASA,
are designed to communicate the quality of the cost
product and its fitness for use.
Application to Cost
In theory, acquisition program managers generally
prefer to seek technologies at TRL 6 or higher;
however, in practice, this is not always possible and
consequently less mature technologies are selected for
development. Cost analysis and support are requested
by program managers throughout all phases of the
acquisition timeline. Therefore, to increase the accuracy
of cost estimates early in the acquisition cycle and
reduce the variance of the total cost estimate; further
research should be conducted where cost intersects
with TRL, IRL, SRL, MRL and/or CRL. Research that
statistically describes the relationship between cost
and the various readiness levels will enable cost analysts
to reduce the variance of the total cost estimate
and provide program managers with more precise
cost figures upon which to base decisions.
Extension of Current Work
Limited studies, research, and databases exist on
relating cost-to-readiness levels.8 The TRL calculator,
developed by the U.S. Air Force, provides a snapshot
of maturity at a given point in time. The technology
program management model, developed by the U.S.
Army, is used to help with technology transition for
program managers. In order to characterize readiness
levels in terms of cost, further research must
be conducted, and the notion that cost is a function
of maturity and risk, transition time (duration) and
acquisition phase must be tested.
Conclusion
Moving to an I&I based acquisition process has
many advantages to the warfighter. The increased
focus on missions allows decision makers and
resource sponsors to focus limited resources on targeted
improvements or acquisitions. Accurate cost
estimates are the keys that unlock successful trade
offs. Cost analysis from the I&I perspective presents
unique challenges and opportunities. Most of
the challenges arise from the breadth of platforms,
systems or capabilities that may be considered for
a particular mission. This necessitates a network of
technical SMEs and cost analysts to cover the requisite
knowledge and skills. The opportunity related to mission-
focused cost analysis is the ability to efficiently
characterize alternatives in order to attain the optimal
benefit for the least cost. Again, this requires accurate,
defendable cost estimates. Overcoming these challenges
while producing accurate estimates will require
some ingenuity and cost research. Fortunately, there
are several promising areas of research, including
messages and readiness levels, awaiting interest and
resources.