By John H. Mayer
Demand for embedded real-time capability in military and aerospace systems is hardly relatively staid and predictable as it once was. While designers today are demanding new features and increased performance, applications are growing dramatically more complex as 8- and 16-bit microprocessors are quickly being displaced by highly integrated, powerful 32-bit CPUs.
At the same time, budget constraints are forcing design teams to discover new ways to lower project costs while increasing competition exerts continuing pressure on project managers to move new designs out of the laboratory and onto the market.
Together these trends are forcing military and aerospace designers to reexamine their strategy for competing in the commercial off-the-shelf (COTS) world. In more cases than not, system manufacturers are deciding it`s time for in-house real-time operating system (RTOS) development efforts to head out the door.
"The days of rolling your own are gone," observes Arlan Pool, manager of advanced technology for Mercury Computer Systems, Inc. in Chelmsford, Mass. "There`s now so many good kernel options it`s hard to justify the effort."
Market pressures are also to blame. "There`s a growing realization that the time-to-market window is just too critical to go off and try and build you`re own custom kernel these days," argues Tom Barrett, president of Embedded Systems Products in Houston and developer of the multi-tasking kernel RTXC.
"Companies are starting to figure out that it`s a lot more efficient to go buy something off the shelf because that`s a piece of intellectual property that they don`t really benefit from if they develop it themselves," Barrett says. "Their reward comes from developing the intellectual property dealing with their application, and it`s starting to sink home that they need to do anything they can to improve the odds that they meet their market window in a timely manner."
The final frontier
Certainly many of those same issues drove the engineers at one of the more highly publicized space projects of the year - The Pathfinder mission to Mars. A development of the U.S. National Aeronautics and Space Administration (NASA), the Pathfinder features a control system that has to manage a complex series of functions, from controlling the spacecraft`s trajectory and descent onto the planet, to running the ground operation once it lands.
Operating in an extremely harsh and remote environment, the system had to be highly reliable, meet strict timing requirements, and have efficient software code. Yet severe budget constraints forced NASA designers to develop a low-cost solution on an accelerated schedule. Key to meeting that schedule was the replacement of customized, in-house solutions with COTS technology.
While the designers were able to chose a relatively low-end 80C85 CPU for the Pathfinder rover, the complexity of the mission dictated the selection of a radiation-hardened version of the PowerPC, the IBM RAD-6000 for the Pathfinder lander. For an RTOS, designers chose to forgo a proprietary solution and opted for VxWorks from Wind River Systems in Alameda, Calif.
While the VxWorks proven reliability in a wide variety of military and aerospace applications was a major factor in its selection, other issues had major influence as well. "The fact that the development tools were already running on the PowerPC was a large factor in the selection of VxWorks," says Lloyd Keith, an engineer with the NASA Jet Propulsion Laboratory (JPL) in Pasadena, Calif.
With development of more than 150,000 lines of code taking place on host systems ranging from AIX and SPARC workstations to a 68000-based VME board, portability was a vital concern as well. "It allowed us to utilize low-cost platforms with commercial components while the flight computers were being developed," Keith says. "So we had our development environment in place and software development under way long before the RAD-6000 was available."
The capability that VxWorks offers to designers to make modifications on the fly was another selling point. "We tell our customers that they can connect into the network and if they have problems in their application that crop up later, they can do simple things while the code`s still running like dynamically change the priority or dynamically link and load new objects," says David Larrimore, vice president of marketing at Wind River Systems. "When your target is millions of miles away, that`s an extremely important capability."
Memory protection
While cost and performance constraints may have at one time driven designers away from microprocessors with integrated memory management units (MMUs), those days are rapidly fading. Relentlessly escalating CPU performance, the ability of silicon manufacturers to integrate more and more functions on-chip, and the rapidly increasing complexity of today`s embedded applications all have helped make a memory-protected RTOS a requirement in many military and aerospace designs.
Over the next few years, many industry observers believe integrated MMUs will become a common feature on inexpensive embedded processors. The growing number of applications migrating to high-performance RISC microprocessors such as the PowerPC offers several examples. The tremendous computing power designers have is driving more and more functions from hardware to software.
"It`s not unusual to have half a million lines of C code running on these processors," says Dino Brusco, vice president of marketing for Lynx Real Time Systems in San Jose, Calif. "When you have that much code, you want to support it in a reliable manner."
For such applications, Lynx Real-Time Systems offers LynxOS, a UNIX-compatible, POSIX-compliant, multiprocess and multi-threaded RTOS that is pre-emptible, re-entrant, and compact. With its modular architecture, the OS can be configured as a small kernel and linked with an application to form a ROMable image or configured as a self-hosted development environment featuring a wide array of software development tools, UNIX-compatible utilities, industry-standard networking, a graphic user interface, and a UNIX-like hierarchical file system.
While UNIX/POSIX-compliance has played a major role, a key factor in ability of Lynx officials to win a large number of military and aerospace designs over the last few years has been its MMU support. "When we talked to Boeing [Seattle] and Lockheed Martin [Owego, N.Y.] about the AWACS program, they talked about wanting high availability and reliability," notes Lynx Director of Sales Brian Grega. "What that boiled down to is our utilization of the MMU. It guarantees that that the entire computer isn`t going to crash if one process has a fatal problem, because the system can lock out that process and everything else that stays resident can recover."
LynxOS will run a variety of different pieces in the Airborne Warning and Command System (AWACS) program including the radar detection system. The RTOS is also running the cabin management system in the Boeing 777 airliner, several functions in the highly computerized Crusader self-propelled howitzer, the Land Warrior body-worn computer, and radar systems in the DASA Tornado fighter-bomber.
Heterogeneous designs
New, state-of-the-art radar designs are also presenting new challenges for embedded systems designers. Many demand heterogeneous systems that combine different RISC processors and digital signal processors (DSPs) in a single system to deliver maximum performance per watt per cubic foot.
"What you typically find out is when you develop a very complex radar or signal processing algorithm, different pieces of it a lot of time can be decomposed for parallel processing in different ways," explains Mercury`s Pool. "Typically what you get are sub blocks or sub components of the algorithm that can be best handled in a multiple program, multiple data (MPMD) programming paradigm, and other parts of the algorithm that can be best handled in a channelized type of architecture. So you see a lot of hybrid designs."
The challenge for designers has been to develop applications so to use the processor best suited for each task to make the most of performance and keep the number of processors necessary to a minimum.
"When you bring up the whole idea of heterogeneous processor architectures and the ability to tailor the system to meet the application need, system and hardware engineers will typically say that`s exactly what I need," notes Pool. "But when you turn around and talk to the software engineer the response is typically, `oh no, how am I going to mix and match all these different kernels and disparate programming tools.`"
For designs such as these, Mercury engineers recently announced the MC/OS runtime environment version 4. Designed to extend the scalability and performance of imaging, radar, sonar, video, and other signal processing applications, the MC/OS provides a single-system model through a common and consistent software environment for heterogeneous systems that combine RISC and DSP processors. "What we`ve tried to do is eliminate the learning curve for a different type of kernel for each different type of processor," explains Pool.
To enable designers to migrate portions of an application to the appropriate processor, MC/OS supplies two classes of application program interfaces - one of intraprocessor services and a second for interprocessor services. Programmers can mix and match a variety of programming paradigms including single program multiple data (SPMD), MPMD, dataflow, and parallel.
The MC/OS currently supports systems that combine the Intel i860, Motorola/Apple/IBM PowerPC, and Analog Devices SHARC DSPs. Later in the year Mercury officials plan to add support for the TMS320C80 DSP processor from Texas Instruments of Dallas. MC/OS also supports a standard ANSI-C interface and the POSIX 1003.1c threads programming interfaces.
Mercury officials claim the runtime environment will open up new embedded applications for synthetic aperture radar, space time adaptive processing and foliage penetration. The system has already been designed into new defense radar systems that combine hundreds of PowerPC RISC and SHARC DSP processors.
Bridging the gap
Of course, not every embedded application is large enough to warrant a UNIX-compatible, POSIX-compliant RTOS featuring memory management support and dynamic process generation. PSX from JMI Software Systems, of Spring House, Pa., is targeted at those applications that need more than a bare kernel, but less than full POSIX compliance. PSX offers a small, efficient subset of systems calls that enables designers to use it as a cost-effective development environment before moving on to a dedicated ROM target board.
The product also offers a superset of C Executive, JMI software`s compact, portable kernel. Occupying as little as 5 kilobytes of ROM, C Executive is available on more than 20 processors and features interrupt-driven device drivers, real-time clock support, a fully preemptive prioritized task scheduler, and four methods of inter-task coordination.
Earlier this year JMI engineers added five new system calls to C Executive and one to PSX. "While these new functions aren`t earthshaking, they were done so that later this year we can easily add dynamic thread support in our next revision," says JMI President Ed Rathje.
The kernel is used in a wide variety of aerospace and military applications including a MIPS R3000-based flight test data acquisition system for the Airbus A340 jetliner and several U.S. Navy avionics applications. One of its more interesting design-ins is on a GPS receiver developed at JPL for the GPS/MET satellite that indirectly measures temperature, pressure, and humidity of the atmosphere. The spacecraft is the forerunner of a network of 100 satellites that will measure the atmosphere over the oceans to help airlines reduce the billions of dollars they spend in fuel and time flying around in adverse weather.
FAA certification
Despite growing application complexity, performance and predictability remain key design selection criteria for RTOSs in hard-real-time embedded applications for avionics and defense. One of the leaders in the performance realm over the past decade has been the Versatile Real-time eXecutive (VRTX) from Microtek of Santa Clara, Calif.
In the mid-1980s VRTX32, Microtek`s second-generation product, was the first off-the-shelf RTOS to offer a constant time scheduling/rescheduling run time system. VRTXsa, Microtek`s high-performance offering and the industry`s first preemptive kernel, added significant advancements in predictability and support for priority inheritance.
The RTOS has been implemented in a wide variety of avionics systems, including the flight control computer Automatic Flight Systems on the McDonnell Douglas MD-11 jetliner developed by Honeywell, the flight management computer systems, displays and other avionics on the McDonnell Douglas C-17 jet airlifter, also developed by Honeywell, an auto-landing systems on the McDonnell Douglas AV-8B Harrier jump jet, and on the flight management computer systems deployed on the Boeing 757, 767 and 747/400 jetliners.
Many of these systems, of course, had to be FAA certified. The MD-11 flight controller, for example, had to meet category 1 (flight critical) certification. And, as is common across the industry, the responsibility for meeting certification requirements fell on the system integrator. "Traditionally the industry has taken a hands off approach, so we gave them as much information as we had, but they had to mold it into the information they had to supply to the FAA," noted Jim Ready, chief technology officer for Microtek. "That worked fine but it put the burden for doing all that on the customer."
In an effort to offload that responsibility, Microtek officials are offering all the documentation required to meet DO-178B, Level A. "The requirements for doing level A are stringent and nontrivial," says Chip Downing, Microtek`s director of consulting services business. "Traditionally a lot of customers have invested a lot of money, probably more than it would cost us if we did it, because they don`t have access to all the code and design documents that we have. So we`re building up an entire documentation set for making VRTXsa and other components that we have certifiable to DO-178B Level A."
Downing says the task calls for tracing every line of source code back to a requirement. "There cannot be any dangling requirements or dangling source code," he says. "You have to essentially have 100 percent test code coverage and also have something called modified condition decision coverage as a way of analyzing the Boolean constructs inside your code. There`s no tool to do it, so you have analyze your code by hand."
Providing documents can be key to the systems designer. "In the end you`re presenting the systems integrator with documentation, but the documentation represents substantial amounts of meaningful effort, including testing, requirements traceability, and this Boolean analysis," Ready says.
The customer is still responsible for getting the system certified, says Downing. "By providing all the documentation related to our component, they simply have to test to our interfaces. After that they can focus on their real application, not the run-time system."
The NASA Mars Pathfinder spacecraft (top) and Navy New Attack Submarine (above) both will make extensive use of commercially developed real-time operating systems.
The NASA/JPL Mars rover design attacks costs by using a variety of off-the shelf components, including real-time software
The Predator unmanned aerial vehicle is another military platform set to receive COTS real-time software.
Windows CE - ready for real time?
Initially targeted at small-footprint personal digital assistant applications, Microsoft`s Windows CE has been increasingly touted as an emerging solution for embedded applications.
The announcement from Microsoft officials at the Embedded Systems Conference last March that they intend to push their compact operating system into embedded markets has sparked growing interest across the design community.
But the general consensus is that Windows CE still has a long way to go before it will offer a serious option for designers of hard-embedded applications in military and aerospace.
The kernel`s relatively large memory footprint and slow response time present two serious constraints, say industry observers. But its most imposing limitation, they argue, is its inability to provide true deterministic performance.
But it is still early warns Mike Dexter Smith, president of VenturCOM Inc. in Cambridge, Mass., a developer of tools for Windows NT applications and one of only three U.S. distributors of Windows CE.
Smith argues that CE has many of the characteristics of embedded RTOS and notes the operating system is ROMable, offers fixed priority scheduling, thread priority levels, priority inheritance, and 32 processes across unlimited threads.
Yet he admits it`s missing features such as memory locking, high speed clocks, and timers, and only has seven priority levels.
While he notes much of that will change in the near future, the real interest in CE lies in its ability to mainstream the embedded market. "For all their warts, CE and NT are mainstreaming this industry," he argues, "because at the end of the day, people want NT applications."
The prospects of an NT-compliant real-time operating system are formidable. Designers of embedded systems would be hard pressed to ignore an RTOS that offers access to thousands of inexpensive tools, drivers for virtually any device, a familiar graphic user interface and application program interface (API), and dramatically low relative pricing.
"I recently looked up the hardware compatibility list for network LAN adapters and found that NT supports 230 different ones," Smith emphasizes.
Those issues haven`t been lost on traditional RTOS vendors. Some, such as Wind River Systems of Alameda, Calif., QNX Software Systems Ltd. Of Kanata, Ontario, and PharLap Software, Inc. of Cambridge, Mass., have brought tools to market over the last year that enable developers to use the Win32 API to access the many tools available while still meeting real-time requirements. And engineers at Radisys Corp. of Hillsboro, Ore., have developed an extension to NT that purportedly adds the real-time response needed for some mission-critical applications.
With his company`s sales growing at over 300 percent each year, Smith says he is relentlessly optimistic. "We`ve already got over 70 design-ins over the last year ranging from flight simulators to industrial control and the interest is overwhelming," he says.
But long term, the prospects for Windows CE in hard-embedded applications may be dictated more by business than technical issues. "Because the applications are mission critical and have such a long life, many of the system developers in the mil/aero arena are required by their customers (i.e. The services) to own source code to the product and Microsoft will not sell source code," notes Brian Weinberg, senior technologist with Lynx Real Time Systems of San Jose, Calif. "Moreover, they offer no fixed revision support and that`s a killer for long term projects."
"Ultimately, it comes down to business focus," adds Dino Brusco, vice president of marketing at Lynx. "The product isn`t focused on the deeply embedded market, not because it couldn`t be, but because that`s not where Microsoft really wants to go." -J.M.
