Air university

As a prelude to seamless verification of the total system, subsystems needed to be tested in preparation for LRIP. In anticipation of the type of environment this new weapon was going to be flown, many tests were proposed that Colonel McClendon and the SPO could not afford. He consistently demanded that contractors do sensible component tests.

Colonel McClendon and Mr. Wasylczyk were frustrated that previous military standard (milstd) tests did not simulate the real world and invariably, a component would fail in theater. On one occasion which set the tone for the rest of the program; Mike Wasylczyk, the lead SDB airframe engineer, needed to test the plastic arming generator (AG) cover that would deploy in flight. Mr. Wasylczk had such confidence in the product’s durability that he hammered on the cover during one SPO meeting to show it would not break. Colonel McClendon approved of Mr. Wasylczk’s methods, accepted the AG cover as designed and dubbed it Trunk Monkey Certified (TMC).¹²⁶ In actuality, TMC testing went above and beyond milstd tests on many occasions.

TMC became more sophisticated when Mr. Wasylczk tested the ball screw on the MBDA diamondback wing. When the wings deployed, they would pivot around the ball screw. Instead of figuring out a milstd test for sand grains per a volume of air to mimic use in a desert environment, Mr. Wasylczk developed an alternative test. The weapon was attached to a jeep and driven around the desert. The wings deployed properly—again, his innovative test worked. These approaches were cost effective, yet when AAC leadership was briefed, they challenged the results because a milstd test was not used. Colonel McClendon was willing to stake his reputation on the improvised testing and asked if the leadership wanted to approve an additional test that would jeopardize “RAA in 2006”--they backed down.¹²⁷

Colonel McClendon created, and the DoD approved, the SDB Single Acquisition Management Plan (SAMP) version for SDD in the middle of CAD in 2003. Just like in JDAM, the SAMP authorized the military SDB PM to deviate from Federal Acquisition Regulations and from Defense, Air Force, and Air Force Material Command (now reformed into Air Force Systems Command) regulations. The SAMP encouraged commercial practices and parts and purposely did not mandate use of military standards for testing. The SAMP called for early contractor involvement, full-time SPO “helpers” assigned to each competitor, and the formation of temporary working groups to assist competitors. The source selection was transparent with open communication, oral proposals, and down-select criteria listed out of the SAMP: Risk (weapons and carriage system design, production representative hardware, aircraft integration, accuracy, effectiveness), Affordability (AUPP weapons, AUPP carriage, SDD contract price), Contractor Past Performance (responsiveness, meeting the IMP/IMS, systems engineering), Mission Capability (weapons and carriage system design, accuracy, effectiveness, growth for precision against movers, in SDB-II), and employing small disadvantaged businesses (pass/fail).¹²⁸ Demonstrated performance of production representative hardware was the key discriminator between the competitors. Ms. Stokley explained:

Colonel McClendon and his Source Selection Evaluation Team (SSET) implemented a new scale of 1-10 to measure system design maturity (this scale is roughly analogous to the Technology Readiness Levels for component technologies defined in the Defense Acquisition Guidebook). This moved the SSET from difficult-to-discriminate low/moderate/high ratings to very objective 1-10 assessments of exactly what had been flown and to what result. Motivated by this, both contractors accomplished a robust Critical Design Review (CDR) during the CAD phase.¹²⁹

Another key feature of the source selection was the use of interim feedback to both contractors during the competition. This is what made it a “rolling” down-select. Three formal evaluation periods were scheduled to include routine and interim feedback for each period. Each evaluation rated risk, affordability, contractor past performance, and mission capability. Except for contractor past performance, interim ratings did not count in the final grade, but provided the contractor with a vector check to make mid-course corrections where needed. Previously, Colonel McClendon recalled contractors spending hours creating “Why I am BLUE” briefs, referencing to the best color grade a company could receive. Furthermore, he felt his SSET was initially lazy and needed to be trained to give feedback information early in the competition and to use data to back it up. The SPO gave contractors a draft feedback a week early to comment on it and gave it back again with a possible.¹³⁰ To enter SDD, level three build-to drawings had to be at least 85 percent complete and the contractor’s Configuration Control Board had to be established and functioning.¹³¹ Following the third and final formal evaluation period, the final selection was made to Boeing.¹³²

Since results-based systems engineering in terms of production representative hardware was so important, contractors received credit if they demonstrated performance greater than the threshold. They did not receive credit, however, for promising greater performance in future spirals. Carefully designed criteria made difficult for any company to gain competitive advantage by under-bidding the work. This strategy allowed the SPO to avoid cost overruns during SDD and/or price hikes during production. Ms. Stokley wrote:

Their estimates had to fall within 15 percent of our government estimate for SDD. For production, we required the contractors to offer fixed prices for lots 1 and 2 and price commitments for lots 3 through 7, and we did independent cost estimates of each contractor’s design. Their bid had to fall within 15 percent of our government estimate of their design costs to be adjudicated realistic by our SSET. Colonel McClendon held another 5 percent, and the AF gave the program about $2M each year of additional funds for further risk reduction activities. It is important to note that we received this additional funding because of our stellar financial execution.¹³³

In August 2003, the CAD competitive risk reduction program ended. After a favorable Milestone B Acquisition Decision Memorandum designating SDB an ACAT 1D program in 2003, the SPO awarded Boeing a three year SSD contract worth $188 million with options to build 24,000 weapons and 2,000 carriages over the next 10 years with a potential program value of $2.5 billion.¹³⁴

SDD Investment and Production Challenges

Systems Engineering (SE) was gaining increased emphasis during the SDB Systems Development and Demonstration (SDD). SE emphasizes the development and organization of complex systems. The technique involves taking a holistic view in the development effort that helps meld all technical contributors into a unified team effort, forming a structured development process that proceeds from concept to production to operation and, in some cases, through to termination and disposal. Some people view SE as writing reports, developing computer models, and conducting virtual simulations. SE is about applying discipline to the processes to get the warfighter what he wants and needs. The SPO measured SE effectiveness by the quality of the product produced, test successes, and changes required versus the volume of paper to accomplish it. Although SE may be highly effective in creating the latest technology breakthroughs for modeling and simulation; the process takes longer and costs more. This section examines how Time Certain Development (TCD) was supported, Seamless Verification (SV) was used, and what to do when the operational environment brings up challenges during production.

The CSAF’s directive of “RAA in 2006” for SDB was in the interest of time. Most trained acquisition professionals believe that a PM needs to balance the three legged stool of schedule, cost, and technical performance to achieve success. This does not mean all three legs have to be equal. Indeed it means that if the PM chooses one leg over the other two, the PM needs to be aware that this will affect the other two legs. If a program is based on schedule, then someone needs to have the authority to flex funding or requirements.

To ensure “RAA in 2006,” Colonel McClendon wanted to do as much component and total system testing up front as possible. By finding money every year and using end of year fallout money, Colonel McClendon was able to buy more spare parts for testing and to buy down more risk. He consistently positioned Boeing and the SPO to be ready for another flight test. In fact, Colonel McClendon warned his bosses not to expect money back from the SDB program and that he would scavenge for other SPO’s unspent money.¹³⁵ Mr. Jaspering was always pleased with Colonel McClendon’s efforts and did not want to be in the situation of “never having enough money to do things right the first time, but always having enough money to do it over.”¹³⁶

What normally took the weapons acquisition community 13 years to accomplish took the SDB program under six years. In explaining Time Certain Development, an anonymous engineer once said, “SE won’t solve what building and testing will!” Figure 4.4 shows what the SDB program accomplished in 63 weeks in what normally takes double the time.

Previous to seamless verification (SV) in the Air Force test world, the weapons acquisition community allowed the contractor to develop the flight test requirements, then go into development test (DT) supported by Eglin AFB, followed by OT supported by Nellis AFB, and concluded with a graduation exercise overseen by Air Force Operation Test and Evaluation Command (AFOTEC) at Kirtland AFB.¹⁴¹ Under SV, DT and OT are involved at the start of the development, live fire, and operational testing per a Test and Evaluation Master Plan (TEMP)--all for the purpose of maximizing the elimination of the bugs in order to get the best weapons out to the warfighter as quickly as possible. SDD used the F-15E as the threshold aircraft in SDD for SV because of compatible Operational Flight Program (OFP) update schedules, current combat role, and availability to support developmental and operational testing.¹⁴² As a pathfinder program:

Some members of the OT community fully embraced the SV initiative. AFOTEC assigned a representative to the SDB SPO. The actual operators from the 53rd Test Wing became involved in the program to provide direct feedback and advice on the feasibility of requirements implementation. And even the DoD Live Fire test community embraced the SV initiative. Their goal was to have no unique live fire testing.¹⁴³

Since a typical guided SDB-I flight test cost $125,000, Colonel McClendon was adamant about being the responsible test officer (RTO) in control. Since Colonel McClendon worked hard for every program dollar and was responsible for schedule, cost, and technical performance, he felt he should make the calls for test pass or fail--and he did. He established very specific primary and secondary objectives. Both Colonel McClendon and Mr. Jaspering signed the test objectives before each test as there would be no confusion post test as to what was expected. In Colonel McClendon’s eyes, the flight test could fail secondary objectives but move on with a pass and with, more importantly, new data. SDB was certified for OT in September 2005, 23 months after the SDD award and after passing 35 of 37 developmental test flights.¹⁴⁴ Colonel McClendon felt that “although the RTO was the decision maker, the SPO had a great relationship with the test wing commander and we ensured we recognized the test wing team that helped us at every opportunity.”¹⁴⁵ Figure 4.5 represents the responsibility Colonel McClendon had in reducing risk with the test opportunities he had.

Indeed, the SDB team fulfilled the CSAF’s directive of “RAA in 2006” by delivering the first SDB weapons and carriages to the Air Force in August 2006.¹⁴⁶ The SDB program also declared Initial Operational Capability on F-15E six months early in October 2006, as well as it being used in combat by a Lakenheath AFB F-15E for the first time in Operational Enduring Freedom.¹⁴⁷ In December 2006, Boeing and the SPO agreed to the first full-rate production contract worth $80 million for 1,600 weapons, 300 carriages and associated support equipment.¹⁴⁸

Finally, this string of accomplishments led to the SDB receiving the 2007 William J. Perry Award by the Precision Strike Association in February 2007 to recognize outstanding leadership and technical achievements resulting in significant contribution to precision strike systems.¹⁴⁹ A year later in February 2008, Boeing delivered its 1000^th SDB-I and 200^th BRU-61 while keeping the program 20 percent under the cost that the SPO has committed.¹⁵⁰



Figure 4.5 SDB Risk versus Test¹⁵¹

“SDB enjoys a great reputation within DoD as a model weapon-acquisition program,” Colonel Richard Justice, commander of the 918^th Armament Systems Group at Eglin AFB, said, “And a significant reason for that is from our strong partnership with Boeing.”¹⁵² The program has not, however, gone without challenges. Every program experiences flight test failures. Luckily, flight test failures have been few and far between for the SDB program. When failure does occur, the SPO and Boeing aggressively assess and mitigate any issues.

The combat environment concerns Mr. Brooks, the current Boeing SDB PM, the most, “The real world will find additional imperfections and the team has to be ready to respond as fast as possible.¹⁵³ Mr. Jaspering, the original Boeing SDB PM, predicts “DoD needs to understand if the test environment is different than the AOR, because some issues won’t be found until deployed.”¹⁵⁴ Then success becomes a matter of reputation. Lieutenant Colonel Chris Baird, commander of the 681^st Armament Systems Squadron for SDB I, claims:

The SDB program has become a victim of its own success following JDAM and the SDB weapon. However, even though the BRUs are contractually meeting their specification in theater, it is below the warfighter’s expectations. Thankfully, Boeing and us can talk candidly with each other and have been working through these issues.¹⁵⁵

There were two BRU-61 anomalies in OT, but due to limited number of test opportunities and the success of the BRU earlier in SV, no one was sure if this was indicative of future issues or not. By being proactive, the SPO and ACC put in place a Carriage Reliability Confidence Program to perform additional testing and to closely monitor carriage failures in the field. Boeing learned things from the warfighters on real world deployments and the changes that had to be made. Unfortunately, improvements will always lag the findings and the warfighter’s patience is short. In retrospect, Colonel McClendon wished he had “trunk monkey certified” the BRU-61 some more.

Mr. Jaspering agrees and supports the current overlapping of SDD with production as depicted in Figures 4.2 and 4.5 because there needs to be an initial funding mechanism for reliability and growth challenges. By keeping SDD contracts in place into the production phase for a few years, the SDB team can correct and understand any issues the warfighter experiences. Again, with SDD overlapping production, the SDB software could be easily modified to handle the close air support (CAS) environment that is more prevalent in Operations Enduring Freedom and Iraqi Freedom as opposed to a stand-off weapon as was originally designed. In the end, ensuring effective communication is one of the most difficult challenges. Unfortunately since the SDB has hit FRP, the SPO and Boeing is coming up with CAS OFP ideas and not the warfighter. Additionally once in production, SV becomes too cumbersome for relatively simple OFP changes.

Despite any past or current challenges, the SDB program has been successful because it avoided the seven deadly sins of acquisition and it met a majority of expectations as explained in Chapter 2. First, choosing the right person, Colonel McClendon, as the SPO SDM PM meant the project had a leader with highly effective people skills. Second, technology was matured by the MMTD and the AFRL. Third, requirements remained stable with just two KPPs. Just like in JDAM, competing contractors’ threshold objectives could be traded for time. Fourth, risk reducing repeatable process came in the form of early testing (trunk monkey certification) of the components and seamless verification of the whole weapon system. Fifth, IPTs not only fostered communication for the supply chain again, but also for decision makers inside the Pentagon. Sixth, CAD had Boeing and LM competing in a rolling down-select and then there was a virtual competition between Boeing and one of its suppliers. Seventh, result based incentives were used for schedule and stability to meet “RAA in 2006.” Expectation-wise, the SDB program met General Jumpers’s target date, provided a warranty, and exceeded accuracy performance. Time will tell if the warfighter totally buys into the weapon.