Home : Home : Warfare Centers : NSWC Dahlgren : Dahlgren Resources : Leading Edge : I&I Leading Edge : Brown and Lloyd
I&I From a Cost Perspective
________________________________________________________

By Jonathan Brown and Sara Lloyd

In today’s budget climate, how can the Department of Defense (DoD) ensure they are still developing and procuring the equipment that our warfighters need? By focusing on specific mission areas and looking across the Doctrine, Organization, Training, Materiel, Leadership and Education, Personnel and Facilities (DOTMLPF) spectrum for solutions to address mission capability gaps, Integration and Interoperability (I&I) is uniquely suited to find the “knee in the curve” and maximize the return on investment. A key component to this optimization concept is the ability to accurately predict costs across a spectrum of mission solutions. An accurate prediction of costs enables decision makers to efficiently utilize their resources to attain the maximum performance for the minimal cost, identify potential trade-offs, and identify areas for cost reduction and/or avoidance. This article discusses the role of cost estimating in the I&I process, including the challenges of mission-level cost estimates. It also provides an evaluation of current cost estimating techniques that may be applicable to mission-level analysis. Finally, it explores promising areas of research that may mitigate some of the current cost estimating challenges, such as the potential for using existing I&I artifacts, technical metrics generated during the I&I process, and Integrated Readiness Level (IRL) as metrics for cost assessments. The integration of cost analysis into the I&I process combines research and development with operational support and readiness in order to provide cutting edge cost support to the warfighter.

Where is cost involved in I&I?

Figure 1 depicts the current I&I process. As previously discussed in this edition of Leading Edge, the Integrated Capability Package (ICP) prioritizes the set of DOTMLPF solutions proposed to cover the identified capability gap or set of gaps. Cost estimates are required for trades studies and analysis of alternatives as well as for each proposed solution in the set. Having estimates of the required funding for each solution allows decision makers to make more informed decisions. As illustrated in Figure 2, tradeoffs between cost, technical, and schedule are required when faced with limited resources.

Challenges of I&I Cost Analysis

Cost Growth

While there are many different ways to measure the extent and nature of cost growth, there are multiple possible explanations and potential solutions for improvement. Often, technology development and acquisition programs are launched with too little focus on system and mission-level requirements needed to develop realistic cost estimates. Program managers may have optimistic schedules and assumptions, such as stable requirements, with corresponding low initial cost estimates that are appealing to funding sources. Realistically, however, significant requirements volatility is common during the early acquisition phase, which in turn causes significant cost growth. When attention isn’t given to possible changes and growth in requirements, budget overruns occur and programs risk being canceled. In order to prevent programs from being canceled, one must acknowledge and plan for the likelihood that requirements will grow and change. Canceled programs consume resources with little or no return on investment; moreover, they represent lost opportunities for viable programs that may also fail due to lack of funding. The result is a replicating program failure effect. The longer the program continues before cancellation, the greater the effect.

Requirements and Data

Squeaky wheels get the grease, and dedicated, tenacious program managers are more likely to receive program funding. Cost analysts rely on these system- specific champions to help overcome obstacles to prioritizing the cost team’s technical or data requests. Costs cannot be determined for requirements that are not defined in technical terms or do not indicate the level of effort required. Therefore, cost analysts depend upon obtaining data from the engineering and technical teams to ascertain what costs will be affected by the various alternatives. Most important to the I&I efforts is to obtain candid technical assessments of impact to other systems to determine the causal effects on the cost. A strong program manager will set priorities that ensure the engineering and technical teams are thorough and responsive to the needs of the cost analyst’s data requests.

Program Focused

Generally, cost estimates, similar to most other systems engineering tasks, are program focused. Cost estimating within the context of the I&I process requires a paradigm shift from program focused estimates to mission focused estimates in order to accurately predict costs across a spectrum of mission solutions. When implemented, this type of analysis is similar to a large-scale Analysis of Alternatives and requires the cost analyst to have breadth vs. depth of analysis. Typically, cost analysis requires some level of specialization. Successful cost analysts often focus on one specific group of systems or platforms or one area of cost analysis. Examples of specializations include combat systems, ships, submarines, rotary wing aircraft, etc. Often, the specific system or platform is large; therefore the cost analyst gains a breadth of experience in a multitude of subsystems that exist within this area of specialization. Similar to the I&I technical subject matter experts, I&I cost analysts need to have not only a broad range of experience but also a large network of experts to leverage when required.

Subject Matter Experts

Expert opinion, or subject matter expert (SME) input, is important to cost estimating by bringing a wealth of experience and knowledge to the estimating process. Many times, experts help to identify analogous systems or provide data on labor hours or testing requirements to build, operate, and maintain a system. At the very least, an expert can provide his or her opinion on cost drivers, functional form of a regression and engineering rules-of-thumb on which the cost analyst can consider in the estimate. When working across multiple platforms, systems, and capabilities, it is not possible to have a dedicated SME for each potential solution on the team as compared to a typical system acquisition cost IPT with dedicated resources. A dedicated team with a robust network of SMEs is needed to have access to the required technical information, similar to a technical warrant holder pyramid as depicted in Figure 3.

Opportunities for I&I Cost Analysis

Despite multiple challenges presented by integrating cost analysis into the I&I process, many opportunities exist. I&I related research will set the conditions for providing more accurate cost estimates early in the acquisition process (pre-Milestone and Milestone A). Mission-level analysis reduces uncertainty, provides better metrics and allows for a more accurate cost model; hence, reducing the overall cost and increasing the probability of program success. Current cost estimating practices include providing cost estimates to program management throughout each phase of the acquisition life cycle. Cost estimates are often required at the earliest stages of a program’s development, i.e., Milestone A or earlier, when the acquisition strategy and requirements are loosely defined. This yields rough order of magnitude cost studies, surrounded by a high degree of uncertainty upon which important program decisions are made. Furthermore, sometimes program-specific analysis focuses on a single platform and a single system. This may lead to a product based on a limited sight picture rather than providing the customer with a holistic, system-of-systems life-cycle analysis that is important to providing the decision maker with various program options (both technical and cost). This program-specific analysis increases the risk that problems will arise when the system or capability must interact and interoperate with other systems to accomplish its mission. Interoperability problems are becoming a focus area and are expensive to fix once the system or capability has been developed. An example of this is the recent Accelerated Mid-Term Interoperability Improvement Project (AMIIP) that was created to address Fleet concerns about force-level interoperability; specifically to address consistent correlation between Cooperative Engagement Capability (CEC) and Tactical Data Link (TDL) tracks.2 Mission-level analysis helps to avoid these interoperability issues and escalating costs by addressing system interactions required to accomplish the mission early, at the concept phase, in an acquisition program, thus reducing potentially expensive interoperability “fixes” in the future.

Another acquisition problem currently facing the Navy is the development of technologies or capabilities with no clear insertion point into the Fleet. These “orphaned” programs have already sunk the cost of development but provide no direct benefit to the warfighter because they are never integrated into a platform. More likely, they are installed onto a platform but not integrated into the combat system. The overall I&I process would reduce orphaned programs by either identifying a specific mission gap each capability will be addressing or, if no mission need or no better solution is identified, the program will not be recommended.

The final opportunity to consider relates to the technical data available for cost estimation. The I&I process puts an emphasis on upfront mission architecture. This architecture process produces metrics that have the potential to be used for cost analysis. The emphasis on upfront mission architecture in the I&I process ensures that the metrics are received early often. This topic will be expanded on later in this article, but some technical metrics that potentially influence cost are information exchange messages and Technology Readiness Levels (TRLs).

Evaluation of Current Cost Process Suitability to I&I Cost Analysis

The following section summarizes the current cost analysis process and identifies the suitability to its applicability to the I&I process. Figure 4 represents the Government Accountability Office’s (GAO) cost estimating process. Often, cost analysts apply a tailored version of the GAO’s Cost Estimating Process that is most applicable to the program and associated tasks.

Define the Estimate and Develop the Estimating Plan

                       

 

Estimating by
 ANALOGY

Estimating by
 PARAMETRICS

Estimating by
ENGINEERING

Estimating by
 ACTUAL COSTS

 What is it?  Single value from single data point Measure of trends across programs Detailed build-up of Lab, Mtl & OH $ An extrapolation of current program cost
 When used? Early in Program Pre-Milestone A &
Milestone A
Milestone B Late in Program Milestone C & LRIP  LRIP
 Full Rate Production
 How is it done? Adjust analogous system cost or create cost factor Apply statistical methods to cost of 2 or more systems (i.e., Develop a CER) Estimate at lowest cost level & sum costs by WBS Use trend from your current contract to estimate your final system $
 PROS Fast, inexpensive
 
Easy to change
Based on >1 data point=>less risky

Can measure error

Easy for what-ifs
More details enables better visibility into cost drivers

Most costs are known
 
CAIG prefers over other methods

CONS 

Based on single historical data point
+>risky!

Trends to be more subjective
Constrained by amount & quality of data (GIGO)

Statistics can be misleading

Labor Intensive

Slow, Expensive

Can lose sight of "big picture" 

Usually too late to use actual costs to adjust or build budget

Not a 1:1 correlation of prototype-to-production costs
                       

Table 1. Cost Estimating Methodologies4

One of the most critical first steps is to define the scope of the cost estimating effort and to set the due date for the completed cost estimate. Upfront planning will still be required for any cost estimate, I&I or otherwise.

Define the Program

I&I cost estimates will require the analyst to define the program. This will be more difficult for an I&I program because instead of a single program, system or capability, the I&I solutions will likely be a range of options to address the given mission. For traditional acquisition programs, this defining information comes from a Cost Analysis Requirements Description (CARD) or other written documentation from the program office. When the CARD is unavailable or does not include enough information, the cost analyst and technical team must work together to identify constraints and reasonable assumptions that will define the effort. For I&I, similar information is still required; however, there will no longer be a single source document or program office. Furthermore, it is unlikely that the depth of information required in a CARD will be available for each option. Similar to, but broader than an AoA, a prioritized list of information required for an I&I cost estimate must be provided by a network of program offices and SMEs. The list of information required could be far more extensive than other early estimation efforts.

Determine the Estimating Structure

An I&I cost estimate will still require the development of a Work Breakdown Structure and identification of an estimating method for each. This part of the process will likely be similar to other cost estimates.

Identify Ground Rules & Assumptions (GR&A)

While the actual ground Rules and Assumptions (GR&A) developed for an I&I cost estimate may vary from a traditional cost estimate, the process to identify them will be similar.

Obtain The Data

Data gathering is the most difficult task in the cost estimating process. Data collected falls into two basic categories: data from previous programs used as a basis for cost relationships and technical data for the current program used to project future costs from the historical actuals. For a typical cost estimate, the analyst must identify SMEs, analogous systems, previous cost estimates on the topic, relevant databases, Cost Analysis Requirements Description, and pertinent program and engineering data from the program office. As mentioned above, for I&I cost estimates, this process is difficult since there are multiple platforms and systems involved to address a specific mission. However, there are potential new sources of technical data available from the I&I mission architecture to help address this complexity.

Develop the Point Estimate

There are four accepted methods by which cost estimates are generated: analogy, parametric, engineering, and actuals. Table 1 summarizes the four methods.

Analogy

Analogy entails using a single comparison value from a system with similar characteristics. This method is most often used early in the acquisition phase when there is little or no data available on the new system. While it is relatively quick and easy to use compared to with other methods, the results may be generalized and subjective.

Parametrics

The parametric approach uses statistical means to measure trends across multiple programs. This method requires a large amount of data; however, the outcome yields measurable and tractable results.

Engineering Estimate

An engineering estimate is a detailed, bottom-up approach which is usually used later in the acquisition process. Gathering the data required can be a slow process that is labor intensive but costs are estimated at a high level of detail which will provide better insight into cost drivers.

Actual Costs

Actual costs or “actuals” are an extrapolation from the current data of the costs that have been incurred. These costs are usually the best and most precise costs; however, often the data becomes available too late in the acquisition cycle to be useful.

While the actual analogies or technical metrics used in a parametric estimate may differ from a traditional cost estimate to an I&I cost estimate, there are no expected variances to the four acceptable methods. Given the information required to use an engineering buildup or actual cost, employing these methods will not likely be frequently used in the I&I process. The fact that the I&I analysis is conducted early in the acquisition life cycle will necessitate a larger use of analogies and parametrics. However, this is not absolute. For example, if an existing technology is proposed as a solution in an ICP, the actual costs of that item would then be used in the I&I cost estimate.

Final Steps in the Estimating Process

The final steps, shown in Figure 4, of uncertainty analysis, presentation to management, and updating the estimate will require no anticipated changes from the traditional process. However, there is a subtlety to be emphasized. A typical cost estimate for a program or system usually resides under a single System Command (SYSCOM) and program office. There may be additional stakeholders, but the SYSCOM and program office are responsible for providing the final review of the cost (SYSCOM Cost Organization) and technical (program office) assumptions and methodologies. By contrast, an I&I cost estimate could include multiple programs or systems that potentially cross multiple SYSCOMs (NAVSEA, NAVAIR, MARCORSYSCOM, etc.). Who would provide the final review and be responsible for the validity of an estimate that crosses multiple SYSCOMs and program offices? Some options for answering this question could include designating a lead SYSCOM for each ICP. The ICP lead SYSCOM would be responsible for approving the entire package. An alternate approach would be to require individual SYSCOM sign off on the respective technical area of a single ICP. The final process is yet to be determined, but responsibility of the I&I cost and technical review will need to be established.

Cutting Edge Research and Promising Topics

Cost estimates for I&I products present several challenges to the cost estimator. These challenges also open up new cost research areas to help mitigate or overcome these challenges. The following topic areas are of particular interest.

Messages, Software Estimating, And Analysis Modeling (SEAM)

A 2009 study conducted at NSWCDD, Software Estimating and Analysis Modeling (SEAM), investigated current software size estimating processes involving Software Lines of Code (SLOC), recommended improvements, and suggested alternatives. One promising alternative technical metric suggested was the measurement of messages. Messages are the packets of information sent from one software component to another. In the research, a preliminary statistical relationship was established between the number of messages and the number of SLOC. While this technical metric for cost measurement requires more research before utilized for cost analysis, it is very promising from an I&I perspective. SLOC is the primary software sizing metric for cost analysis, but as mentioned above, is not likely to be available early on during ICP development. Because of the emphasis on mission architecture, message counts are likely to be available for I&I analysis. If the preliminary results remain stable under further research, this new metric will provide a link between I&I mission architectures, SLOC and cost. To capitalize on this research would require the I&I ICP development process to include sufficient time to conduct the architecture development at a low enough level to be useful to the cost estimate.

Readiness Levels

Previous GAO assessments have correlated low technology maturity with programmatic problems; programs that began development with immature technologies saw a 32% cost growth and 20 month schedule growth.5 There is a growing demand for cost analysis during the early phases of the acquisition cycle when the level of technology maturity is low. Therefore, a promising area of future research, and one that serves to provide significant dividends in the future, includes relating cost to TRL, or any of the various readiness levels, in the face of limited data (see Figure 5).

“Maturing new technology before it is included on a product is perhaps the most important determinant of the success of the eventual product—or weapon system.” The GAO (1999) also encouraged the use of “a disciplined and knowledge based approach of assessing technology maturity, such as TRLs, DoDwide (p. 7).”6

Types of Readiness Levels

There are multiple types of readiness levels that may be relatable to cost. Technology Readiness Level (TRL) is probably the most commonly known readiness level because it is used as the DoD standard evaluation system to categorize hardware, software or system concepts. TRL is simply a measure of an individual technology and it does not account for readiness for use in the system context. The definition of TRL varies across the Department of Defense, Department of Homeland Security, Department of Energy, National Aeronautical Space Administration (NASA), and European Space Agency. Integration Readiness Level (IRL) is the status of the connections between the technologies. System Readiness Level (SRL) is the system-level appraisal of development maturity and is a function of IRL and SRL. Manufacturing readiness levels (MRLs) are quantitative measures used to assess the maturity of a given technology, component or system from a manufacturing perspective. Finally Cost Readiness Levels (CRLs), as used by NASA, are designed to communicate the quality of the cost product and its fitness for use.

Application to Cost

In theory, acquisition program managers generally prefer to seek technologies at TRL 6 or higher; however, in practice, this is not always possible and consequently less mature technologies are selected for development. Cost analysis and support are requested by program managers throughout all phases of the acquisition timeline. Therefore, to increase the accuracy of cost estimates early in the acquisition cycle and reduce the variance of the total cost estimate; further research should be conducted where cost intersects with TRL, IRL, SRL, MRL and/or CRL. Research that statistically describes the relationship between cost and the various readiness levels will enable cost analysts to reduce the variance of the total cost estimate and provide program managers with more precise cost figures upon which to base decisions.

Extension of Current Work

Limited studies, research, and databases exist on relating cost-to-readiness levels.8 The TRL calculator, developed by the U.S. Air Force, provides a snapshot of maturity at a given point in time. The technology program management model, developed by the U.S. Army, is used to help with technology transition for program managers. In order to characterize readiness levels in terms of cost, further research must be conducted, and the notion that cost is a function of maturity and risk, transition time (duration) and acquisition phase must be tested.

Conclusion

Moving to an I&I based acquisition process has many advantages to the warfighter. The increased focus on missions allows decision makers and resource sponsors to focus limited resources on targeted improvements or acquisitions. Accurate cost estimates are the keys that unlock successful trade offs. Cost analysis from the I&I perspective presents unique challenges and opportunities. Most of the challenges arise from the breadth of platforms, systems or capabilities that may be considered for a particular mission. This necessitates a network of technical SMEs and cost analysts to cover the requisite knowledge and skills. The opportunity related to mission- focused cost analysis is the ability to efficiently characterize alternatives in order to attain the optimal benefit for the least cost. Again, this requires accurate, defendable cost estimates. Overcoming these challenges while producing accurate estimates will require some ingenuity and cost research. Fortunately, there are several promising areas of research, including messages and readiness levels, awaiting interest and resources.

Article Images



Figure 1. Schematic of the I&I Process1



Figure 2. Trade-off between Cost, Technical, and Schedule
                


Figure 3. I&I SMEs Require Large
SME Network for Support

  


Figure 4. The Cost Estimating Process3        
        
 

Figure 5. Programmatic Risk as a
Function of Technology Readiness
Level7


Article References

  1. Learn, “I&I Overview”, March 2013.

  2. Bill Williford, Director, Integrated Combat Systems, “Maritime Crossroads: Strategies for Action”, www.navsea.navy.mil/Media/ SAS2013/2.%20PEO%20IWS.pdf, April 2013.

  3. http://www.gao.gov/new.items/d093sp.pdf

  4. https://dap.dau.mil/acquipedia/Pages/ArticleDetails.aspx?aid=8e8f5bf3-f517-4e98-8c07-80bb8670a830

  5. www.dodmrl.com/Gordon_NDIA.ppt

  6. GAO, “Best practices: Better management of technology development can improve weapon systems outcome”, 1999, (GAO/NSIAD-99- 162).

  7. GAO, 1999, p. 24

  8. Studies related to cost and readiness levels: El-Khoury and Kenley, 2012; Lee & Thomas, 2003; Hoy & Hudak, 1994; and Dubos & Saleh, 2008.