Developers harness the latest software engineering tools to deliver reliable, mission-critical, real-time embedded software to mil-aero users.
By Courtney E. Howard
The U.S. Department of Defense and various technology companies specializing in military and aerospace applications endeavor to take advantage of commercial off-the-shelf (COTS) components whenever possible and feasible. When it comes to software development, COTS comes in the form of open-source code, which is readily and freely available.
“The promise of open source is better quality, higher reliability, more flexibility, lower cost, and an end to predatory vendor lock-in,” according to the Open Source Initiative (OSI) in Redwood City, Calif., a non-profit organization that advocates the benefits of open source. The OSI maintains the Open Source Definition (OSD), which involves 10 criteria. In short, the OSD makes the distinction that open-source code should not only be free to acquire, modify, and distribute, but also include the license and right to its use. If a software developer and its clients are not licensed to use the code freely, then it should be categorized as shared source code.
An ever-increasing amount of open-source software components are employed in development projects today, and the use of open source continues to grow, says Bill McQuaide, executive vice president of product development at Black Duck Software in Waltham, Mass.
A 2006 study from Forrester Research Inc. in Cambridge, Mass., reported that 56 percent of North American organizations are already using open-source software, and another 19 percent plan to use it in the coming year. In 2007, LinuxWorld noted: “By 2011, at least 80 percent of commercial software will contain significant amounts of open-source code.”
COTS caveats
Many industry players and end users involved in military and aerospace applications express concern over the increasing use of open source, arguing that embedded software in the commercial/consumer market is entirely different from those in mil-aero environments.
“The distinguishing characteristics for embedded software in the mil-aero market are the strong safety/security requirements and the need to satisfy hard real-time constraints,” explains Robert B. K. Dewar, emeritus professor of Computer Science at New York University, and president and CEO of AdaCore in New York. “These requirements are absolute, and cannot be compromised by time-to-market or enhanced-functionality considerations.” It is unacceptable, he says, to get the latest airplanes in the sky right away if the software fails—and it is a stark contrast with consumer software, where it is normal to compromise reliability in favor of features and rapid time to market. Embedded mil-aero programmers know that not only is reliability possible, but it is also mandatory.
The Boeing 787 Dreamliner’s flight-critical, embedded software is built on the WRS ARINC 653 system and is assembled from software components by multiple subcontractors. (Image courtesy The Boeing Company.)
“The hard-real-time requirements are also unique,” Dewar says. “All operating systems, including consumer systems like Windows, are real-time in the sense that you press a key and expect to have something happen, but it doesn’t matter if it occasionally takes a long time for a response. On an airplane, the pilot presses the thrust lever forward and expects the additional thrust immediately, always.
“All this means that the tools, languages, and operating systems used by mil-aero software engineers have to be suitable for generating absolutely reliable software that can meet the stringent requirements for safety and/or security certification,” Dewar says.
No COTS allowed
The debate over the use of COTS software is not unlike that over COTS hardware. Proponents of open-source code argue the benefits of time- and cost-savings and robust feature sets available via shared software components, whereas opponents find that COTS should never enter into mission-critical software due to an inherent unreliability.
“The normal tools used for the generation of consumer-level software—such as the Microsoft Windows operating system, the full C or C++ languages, and the entire set of Java libraries—are simply not well suited for mil-aero applications because of the requirement for absolute reliability,” explains Dewar. Windows, he notes, is an amazing system with remarkable functionality, but consumers often experience system crashes and problems. “For most consumers, a few mysterious crashes are no big deal, but that attitude is simply unacceptable in the mil-aero domain.” Others are inclined to agree.
“I find it particularly important that no code inserted into the target processor by the development tools be tainted by open source or otherwise be of unknown origin,” says Dr. Ole Oest, chief technical officer at DDC-I Inc. in Phoenix. “Only then can we have reasonable confidence that the software is safe and secure.
PERC Ultra virtual machine technologies from Aonix have been used in Joint Unmanned Combat Air Systems, or J-UCAS. (Image courtesy Northrop Grumman.)
“Naturally, where lives are at stake the application should be functioning correctly from the day it is first fielded,” Oest continues. Embedded software developers can design and implement such software with the use of the best software engineering tools—those that enforce programming language standards, guarantee that a programming language construct will execute as specified by the language designers, and insert checks into the application executable, which in many instances would detect if the executing application is running astray, says Oest.
By selecting the best development tools possible, developers can achieve the best possible quality of the resulting application, such as in the areas of correctness and real-time performance, says Oest. This practice will help ensure that future fix-releases can be avoided or minimized. “It is often costly to replace mil-aero applications after they have been fielded for reasons such as: Volume, the sheer number of systems to be upgraded; and logistics or inaccessibility of the systems, such as satellite or weapons systems deployed overseas. Costly re-certification of parts of or all of the application is another reason to try to get it right the first time.”
For many industry players, suppliers and users alike, reliability begins with the programming language.
Ada acclaim
Specialized tools are more appropriate than consumer-level components to mil-aero applications, says Dewar. As an example, he cites “carefully constructed subsets of well-defined languages, such as the SPARK subset of Ada that lends itself well to formal verification. Historically, the Ada programming community has always had a very strong commitment to quality and zero-defect engineering.”
“A nice illustration of what reliability is about is found in the Boeing 777, which is an all-Ada plane,” notes Dewar. “Well that’s not quite true; the one non-critical component on the plane is the cabin entertainment system, which is written in conventional C++ using typical consumer non-certified approaches, where features are more important than reliability. The 777 may never have crashed due to bugs in the flight-critical Ada software, but the cabin entertainment system crashes and misbehaves with annoying regularity.”
The Java programming language has been adopted by the U.S. Navy’s DD(X) program. In partnership with the U.S. Navy, Northrop Grumman and Raytheon are leading a National Team to design DD(X), a multimission destroyer.
Operation of the new Boeing 787 Dreamliner is controlled by a complex network running a large body of flight-critical, embedded software. This software is constructed on top of the WRS ARINC 653 system, and is assembled from a collection of components by multiple subcontractors, notes Dewar. Much of the software is written in a special Cert subset of Ada, called GNAT Pro High-Integrity Edition for DO-178B, provided by AdaCore as part of the software development package.
To date, no loss of life on a commercial aircraft can be attributed to a software bug, which Dewar considers a remarkable record. “There is still room for improvement,” he says. “We have had some incidents in which software problems have cropped up, but never so far with fatal results.”
Find errors early
Whether open source proponents or opponents, industry software experts do agree that testing is an all-important piece of the development puzzle. It is critical to find errors early. After all, a lot is at stake.
When safety-critical software fails, the consequences can be severe. Paul Anderson, vice president of engineering at GrammaTech Inc. in San Jose, Calif., cites memorable examples: The Patriot missile failure in 1991 caused 28 deaths. The Ariane 5 failure in 1996 resulted in the loss of a $370 million payload. “As software plays a growing role in safety-critical systems, concerns about reliability are increasing,” he explains. “Similarly, security vulnerabilities are a big concern in the defense market. The Titan Rain attacks from 2003 to 2005 resulted in the ex-filtration of terabytes of data from NIPRNet, a U.S. Department of Defense network.”
“The entire tool chain must support the notion of eliminating problems as early as possible,” suggests Dewar. “The programming language must be designed from the start with safety in mind, but without sacrificing real-time predictability or performance. The compilers must detect every possible error as early as possible. The debuggers must handle the complexity of debugging real-time programs. Perhaps most important of all, the programmers must have an orientation to correctness and reliability that overrides all other considerations.”
Some industry pundits find that embedded mil-aero software is far more secure, safe, and high-quality than its consumer-level counterparts. Given the prevalence of safety-critical applications in mil-aero applications, the development process typically is more strict and well documented and testing is more thorough. Moreover, the lifecycle of mil-aero software is much longer than that for consumer applications; fielded systems may be deployed for decades. The development process may take several years and integrate complex components from multiple contractors.
The Patriot missile failure in 1991, which caused 28 deaths, has been attributed to a software problem.
“The need for high reliability combined with long deployments makes software quality extremely important,” Anderson acknowledges. “Systems are growing in complexity, making it more challenging to ensure reliability. As a result, development organizations are augmenting traditional testing approaches with automated software-analysis tools.”
GrammaTech’s CodeSonar static source-code analysis tool is being used by mil-aero companies to improve software quality. CodeSonar performs whole-program analysis and identifies serious bugs, including buffer overruns, null-pointer dereferences, and memory leaks. Static-analysis tools are increasingly being used to find areas where code violates the fundamental rules of the language and where standard libraries are being used in a wrong or risky way, as with race conditions, closing a file twice, or operating on a socket when it is in the wrong state. CodeSonar users include GE Aviation, Lockheed Martin, Northrop Grumman, Japan’s Space Engineering Development Co., other international organizations, and firms developing spacecraft code.
GrammaTech won a $600,000 increase to an existing contract with National Aeronautics and Space Administration (NASA) for developing advanced static-analysis technology. The NASA-funded technology will advance the static-analysis testing of high-assurance software and help NASA increase the reliability of mission-critical software, such as through automated checks for safety-critical systems.
Two approaches are gaining acceptance as partial solutions to the software safety problem, according to Anderson. “The first is the adoption of coding standards, and the second is the use of advanced static-analysis tools to both enforce compliance with the standards and uncover latent programming flaws.” GrammaTech’s work with NASA work combines both of these approaches.
In the first phase of the contract, GrammaTech implemented technology that automatically checks whether software adheres to 10 rules for developing safety-critical software proposed by Gerard Holzmann, technical lead at the NASA/JPL Laboratory for Reliable Software.
When he proposed the rules in 2006, Holzmann remarked: “Tool-based checks are important because manually reviewing the hundreds of thousands of line of code that are written for larger applications is often infeasible.”
COTS and costs
Embedded software employed in today’s mil-aero applications is often composed of millions of lines of code. Some of this code has been developed for the specific mil-aero application, but most of the code is repurposed from the commercial world, explains Kelvin Nilsen, chief technology officer at Aonix North America Inc. in San Diego.
As mil-aero systems grow in size and complexity, Nilsen continues, there is a strong trend toward the increasing use of COTS technologies, especially in software. The reason is largely economic.
“The deployment volume for mil-aero systems is usually much lower than for consumer applications,” Nilsen explains. “This creates an economy in which software development dominates the total project cost. In contrast, for many consumer applications, the deployment volumes are so high that the costs of software development are relatively small in comparison to the total costs of deployment.
Embedded software developers are adopting tools that simplify the use of commercial code in mil-aero applications. Specifically, developers want to ease the processes of: porting code to the appropriate mil-aero platforms, integrating independently developed capabilities into new systems that offer the combined capabilities of the independent modules, and adding to existing COTS software components such aspects of information assurance as real-time, safety, security, and reliability that may not have been present when the software was originally developed for the commercial sector.
Aonix product engineers responded to these requests with a family of Java products designed to enable the use of COTS technologies within the mil-aero domain. Project teams using the company’s real-time Java technologies are roughly twice as productive as C or C++ developers during the development of new capabilities, and are up to 10 times more productive during the maintenance of existing software, says Nilsen.
Maintenance activities applied to existing software include: installing incremental changes to address performance issues or correct operating errors, porting the software to a new CPU architecture or a new real-time operating system (RTOS), and combining the software with other software systems to build integrated solutions with the combined capabilities of independent modules. “Given a particular software-development budget, proper selection of tools and appropriate leverage of off-the-shelf software capabilities will often yield a 10-fold improvement in deployed system capabilities,” Nilsen explains.
COTS software in UAVs
PERC Ultra virtual machine technologies from Aonix have been applied to such large-scale, complex software systems as the Aegis warship software upgrade, J-UCAS unmanned aircraft, and Future Combat Systems. The PERC Ultra product also serves as the foundation upon which engineers at QinetiQ in Mclean, Va., and Farnborough in the U.K will build the Taranis Reasoning Layer of the Autonomy Mission System.
Taranis is to be the largest unmanned aerial vehicle (UAV) built by the United Kingdom Ministry of Defence. The UAV’s capabilities will center on targeting and attack, opposed to surveillance and reconnaissance in current UAV systems.
The Taranis Reasoning Layer software incorporates complex decision-making and optimization algorithms on an embedded processor. It is responsible for making high-level plans which control the flight path and sensors to achieve the UAV’s specific mission.
“We chose to develop the Taranis Reasoning Layer with Java due to the broad range of capabilities of that platform,” Pete Baynham, managing director of QinetiQ’s Command and Intelligence Systems business, explains, “but we needed PERC Ultra to support practical deployment in a real-time, embedded environment.”
Embedded software developers are increasingly adopting component-based development and relying on software supply chains, rather than building entire solutions themselves. COTS software components enable them to: accelerate time to market, improve software quality, expand feature sets, reduce lifecycle cost, and use next-generation architectures.
The use of open-source code brings with it a clear upside, but it is not without a downside as well.
“Enterprises can’t afford to ignore the many benefits of open-source software, but they can’t afford to ignore the risks,” says Joe Niski, analyst with the Burton Group, headquartered in Midvale, Utah. The goal, then, would be to put products or services into place whereby firms can gain the open source benefits and mitigate the associated risks. “Automating the process of selecting and provisioning approved open-source components presents the path of least resistance—they make developers’ jobs easier while providing the auditing and accountability management needs,” Niski adds.
“In the initial phases of the software industry’s uptake of open-source software, we witnessed enthusiasm from software engineers, but caution on the part of development managers because of the policy issues and risk associated with this approach,” Douglas Levin, chief executive of Black Duck Software, mentions. Black Duck Software’s solutions are designed to accelerate developers’ ability to leverage external software, which includes harnessing the benefits of open-source components, while managing the associated risks.
By bringing open-source/external software into a project, developers could potentially be opening themselves up to issues with licensing rights and restrictions, loss of intellectual property, software defects, export regulations, contractual obligations, injunctions, and security vulnerabilites, not to mention escalating support costs.
“If you are building a mission-critical application or a product that will be used in a mission-critical way, you need to support your customer if they have a problem,” says Bill McQuaide, executive vice president of product development at Black Duck Software. “When you bring in external software and something goes wrong, as a development manager, you want to know where to go to get that problem solved and do so in a very timely fashion.” To aid software developers with this and other issues, as well as to accelerate component-based software development in this era of open-source software, the company unveiled Black Duck Code Center.
“It is important to use open source in a safe way,” explains Jim Berets, director of product management at Black Duck Software. “Black Duck Code Center enables components to be put through an approval process.” Black Duck Code Center is a software component selection, approval, and tracking system designed to enable software development organizations to maximize benefits from the reuse of open source and other third-party software components.
Companies are increasingly mixing internally developed code with code from open source communities, partners, vendors, outsourcers, and other third-parties. Black Duck Code Center helps these firms address life cycle challenges associated with application development using internal and externally developed components, also known as mixed-code or mixed-language environments.
Mixing it up
“A large body of legacy software exists within the mil-aero space,” says Oest, “and the programs in this field are typically very long lived.” With long life cycles comes obsolescence issues and an eventual need for modernization, of both hardware and software components.
“In many cases it would be prohibitively expensive to rewrite the applications for the new hardware platform,” Oest admits. “The market seeks tools which would allow for the legacy software to be used basically unchanged and remain in the programming language in which it was originally programmed, say JOVIAL or Ada, whereas enhancements in functionality can be added in a language with which today’s programmers are more familiar, such as Java or C++.”
Executives at a few major defense contractors have contracted DDC-I to help them migrate complex applications to new languages, processors, integrated development environments (IDEs), and run-time environments. DDC-I’s services team currently is working with one defense subcontractor to upgrade the target acquisition and fire control software for a U.S. Army tank. As part of the project, DDC-I engineers will port “bare bones” Ada code originally developed for Motorola 68000 processors and Texas Instruments C40 digital signal processors to a PowerPC running a popular COTS RTOS, all supported under an open IDE. The new software will enable vehicle gunners to detect, identify, and engage tactical targets at greater operational ranges, increasing ground force survivability and target specificity.
Open environment
The market is moving toward an open IDE called Eclipse, in part because it enables easy “mix-and-match” integration and interoperability of software development tools from different vendors, according to Oest. Eclipse “looks and feels” the same regardless of the underlying operating system, such as Windows, Linux, Solaris, etc.; as a result, it eliminates the need to relearn the IDE should the underlying host computer platform change.
DDC-I has debuted Scorpion, an Eclipse-based tool supporting the Real-Time Specification for Java (RTSJ) in accordance with the language definition (JSR-1). Scorpion enables the development of real-time applications in Java. Support for safety-critical application development will be added later this year, when the Safety-Critical Java (SCJ) standard (JSR-302) is finalized.
Scorpion, in contrast to other Java systems, provides real-time garbage collection to help developers meet hard deadlines and enables the programmer to express the entire application in full Java to take advantage of all object-oriented features, even in the real-time portions of the application.
Also, says Oest, “Java programmers are easy to come by, which makes Java a good choice in itself.” Scorpion is compatible with DDC-I’s family of compiler/debuggers for Ada, C, Embedded C++, and FORTRAN. When plugged into DDC-I’s Eclipse-based IDE “OpenArbor,” Scorpion provides programmers with the key benefit of mixed-language development through an easy-to-use wizard that generates the necessary interface code. Legacy code can be left as written in, for example, Ada, and new code can be written in Java and call into the Ada code, Oest continues.
Jump to Java: real or imagined?
Is Java making in-roads in military and aerospace embedded software environments? The industry is rife with conflicting reports and viewpoints.
“Despite some challenges from C++ and Java, Ada is still the technology to beat in high-integrity, real-time military applications,” says Dewar.
At the same time, Dewar acknowledges that many people who would like to see Java applications in mil-aero applications, including aviation. “So far, there is not one line of certified Java in flight, at least according to a major vendor of Java technology—and they should know,” says Dewar. “That’s not at all surprising. Java simply wasn’t designed with these requirements in mind.”
The Java programming language is not yet widely accepted in the field of mil-aero, says Oest. But, he points out, Java is making inroads in the embedded mil-aero software market; in fact, a large military program, the U.S. Navy’s DD(X) destroyer, has adopted Java. “The RTSJ version of Java, and suitable implementations thereof, has finally made Java a viable programming language for hard real-time applications,” says Oest.
SCJ, a subset of RTSJ, will be released later this year and make Java viable for safety-critical applications. After the release of SCJ, Oest predicts increased demand for a secure version of Java or a set of tools that can support the development of secure applications in Java—applications that are hack-proof and tamper-proof.
The broadest adoption of Java to date, says Nilsen, has been in applications that have no real-time requirements. “Many organizations have not yet realized that they can exploit the portability, maintainability, and reuse benefits of Java in applications that have hard or soft real-time constraints,” he says.
All Java is not the same, even within the domain of real-time Java; as a result, it is very important to use the right tool. “Soft real-time Java tools are most appropriate for large, complex, dynamic software systems,” says Nilsen. “Hard real-time Java tools are most appropriate for smaller, simpler, static systems.”
Nilsen and his team at Aonix are responding to a particular need identified by its customers: To extend the benefits of Java into the lower layers of a typical mil-aero application. Traditionally, the lowest software layers in these systems are implemented in C or C++, and the low-layer code integrates with higher-layer Java code using the Java Native Interface (JNI) protocol.
Two significant shortcomings exist with this traditional approach, Nilsen explains. “First, there is a high-performance overhead every time control passes between Java and C. Second, the integration of C code into a Java application compromises the integrity of the Java security model. C programmers, either accidentally or maliciously, are able to examine and modify arbitrary memory addresses that are under control of the Java virtual machine.”
Aonix team members, having supported embedded real-time deployments of Java software for more than a decade, find that the integration of C code into Java applications represents the single most common source of errors. “These errors are among the most difficult to debug, because the problem often manifests itself as a Java application or Java virtual machine bug,” Nilsen notes. “We have responded to this serious problem by introducing a new hard real-time Java technology, which offers the same throughput, memory footprint, and response latencies as C code, but which is much more secure than C code because it preserves the Java security model, including full byte-code verification.” An Aonix demonstration has shown that an all-Java solution ran more than twice as fast as a solution that used a JNI integration of Java and C.
COTS adoption in the embedded mil-aero software segment will continue, and proponents and opponents will co-exist—and no doubt, the military and aerospace industry and end users will be better for it. (Weigh in on this topic in The Mil & Aero Blog, online at http://www.pennwellblogs.com/mae/2008/03/java-anyone.html.)
