Creating operating systems and kernels that are blazingly fast simply does not make the grade today, as systems integrators increasingly demand software that not only runs quickly, but also will not crash, will not damage other programs, and is secure from hackers.
By J.R. Wilson
While the world of embedded real-time operating systems (RTOS) has been growing constantly since the concept was first implemented at the board and chip level, new hardware and software capabilities-and the ability to meet new security, as well as safety-critical certifications-have opened new markets for those willing and able to meet the costs.
The two most important new targets for embedded applications are part of the growing security requirements of the military, homeland security, and some parts of commercial aviation. The first of those is Evaluation Assurance Level 6-augmented (EAL6+); until recently, no operating system had ever been certified beyond EAL5 on that seven-level scale. The other is Multiple Independent Levels of Security (MILS), which allows the mixing of multiple security level applications on one microprocessor.
A major capability boost came with time and memory partitioning, which keeps a rogue application from taking up too much central-processor time and prevents one application from corrupting another’s data.
The various customer communities have also demanded standardized interfaces for portability so they can switch from one vendor to another in the future without scrapping the entire system, as well as for greater graphics capability. The ARINC 653 standard is specialized to time and memory partitioning, while general system OS standards fall under the POSIX Application Programmable Interface (API).
“Architecturally, graphics were outside the purview of embedded operating systems and were a tough capability, especially in certified systems in a deterministic way,” says Greg Gicca, director of safety and security product marketing at Green Hills Software in Santa Barbara, Calif. “But recent announcements from a variety of vendors now offer certified graphics libraries, which is fairly new.”
Cost roadblock
A major roadblock has been cost; it takes $1000 per line of code to certify a MILS kernel compared to $60 to $80 a line for DO-178B-the FAA safety-critical standard for avionics. Even a targeted MILS limitation of 5000 lines of code would be at least twice the cost of DO-178B certification for a full RTOS, which typically is 40,000 to 80,000 lines. Another customer-driven factor has been SWAP-size, weight, and power.
Advancing hardware and software technology has greatly reduced SWAP and made MILS certification possible. At the same time, increasing demand is bringing down the cost of EAL6+ and is expected to begin doing the same for MILS.
“MILS separation kernels-the new term for RTOS in a MILS environment-doesn’t allow leakage of data between the different partitions on a microprocessor,” says Chip Downing, senior industry marketing manager for the aerospace and defense team at Wind River Systems in Alameda, Calif. “You have to do formal analyses to prove your MILS separation kernels, so you try to make the kernel as small as possible to achieve that.
The LynxOS-178 operating system from LynuxWorks is part of the Fire Scout unmanned aerial vehicle, shown above.
“Instead of having three different devices or even three different pieces of equipment, you can mix security levels on one device, from unclassified to Top Secret applications, reducing SWAP in terms of gathering intelligence or transmitting secure communications,” Downing continues. “Most people in the industry expect their initial customers-funding the initial certification-to be large government contractors. So in most cases there won’t be a penalty for subsequent customers, although the costs will remain higher for MILS certification.”
The U.S. military’s commitment to network-centric warfare also has changed the concept of secure operations significantly, from a historic norm of “radio silence” to a new requirement for constant, high-level, bidirectional communications across the battle force.
“The next generation is not just the fact that everything is networked, but the security implications of that. So we’re seeing a requirement for commercial RTOS with high reliability that takes advantage of some of the safety-critical aspects of commercial avionics,” says Robert Day, marketing vice president at LynuxWorks in San Jose, Calif. “The OS requirement also includes a lot of networking middleware over which everything communicates; underlying that is an RTOS. The security factor will be an interesting challenge that also links into homeland security, where the security of information and the credibility of the system links into reliability, but with an extra step in terms of certifying to the new security standards now coming out.”
Classification levels
The advent of the single MMU-partitioned microprocessor and MILS certification has largely addressed that need, which has created a new developer environment for rapid deployment of coresident secure and nonsecure applications. It also eliminates the need for a human with Top Secret clearance to handle all data, effectively raising everything to the highest classification level.
That is especially important in a joint forces or multi-agency environment, where real-time data at all levels must be communicated quickly in an understandable format. Achieving that requires a ground-up architecture designed to separate data by security level and enable access only to those authorized at each level.
Curtiss-Wright Controls Embedded Computing’s SCP/DCP 122 3U cPCI COTS single-board computer used on a European safety certifiable program.
Wind River officials recently introduced their company’s VxWorks Secure Real-Time Operating System (SRTOS), based on MILS concepts and targeted for EAL7 certification, to address this evolving new market. It includes a secure separation kernel, security policy database, middleware layer for drivers, file system, network stack, and other common MILS components.
Building on the VxWorks ARINC 653 product, the SRTOS will use a previously reviewed time and space scheduler to control co-resident multilevel security applications on one microprocessor. It also provides several APIs-low-level Core OS Interface Library (COIL), industry standard interfaces, and middleware APIs-to optimize application performance.
“Today, everyone is looking at the new security standards and integrating those into their embedded RTOS,” Downing says. “I think that will start in avionics, which already is used to stringent safety-critical standards. Working Group 72 at EuroCA, a multination avionics group that includes the U.S., and RTCA, which defined the original FAA standard, are defining standards that integrate safety and security for avionics embedded RTOS. In the past, industrial safety and other industries have looked at avionics as a starting point for safety standards. We expect the same thing will be true for security.”
Secure middleware
The actual MILS certification would apply to the middleware, not the RTOS, and so is not a direct concern of RTOS vendors. However, to maintain market position, they will need to ensure their embedded operating systems can function fully in a MILS environment. Day says LynuxWorks is also working on separation kernels to address the requirement for EAL7 applications and MILS.
“LynxSecure is designed from the ground up to meet the EAL7 security level. It is very efficient, which means it also is relatively small, so you can put other OS on top of it. It will be able to take advantage of new virtualized hardware coming out from companies such as Intel and allow the guest OS to run in both software and hardware virtual partitions,” he says. “The software virtualization maps down on the hardware virtualization to improve efficiency, although the separation kernel will run well on a standard processor, with the separation done in software.”
The software will also run on a multiprocessor. “Because it has hard partitions, you can run guest operating systems that are not running high-security applications, so the single system can do high security, medium security, and low security, all at the same time,” Day says. “Essentially, we believe that is what is needed to meet the MILS security requirements in tomorrow’s high-security systems.”
For applications requiring less than Top Secret security, another Lynx variation will have brickwall partitions instead of MILS-level separation. Day says that will enable LynxOS-SE (security enhanced) to run high-security applications, as well as medium up to EAL4+ without the need for full separation and an EAL7 kernel.
“The military and homeland security are driving a lot of this now, but we expect lots of other applications and industry spaces to look for security in terms of industrial and commercial requirements,” he adds. “That is not necessarily top secret, but enough to stop hackers. There will probably be more of these solutions around because the security and certification requirements are not as intense as the ground-up design we’ve had to do for separation kernels at the EAL7 and MILS level.”
Having an RTOS already certified at EAL6+ and to operate in a MILS environment-and providing the necessary support artifacts-is a major advantage to companies trying to certify their applications or products to those levels.
“Another part of the safety-critical arena is reusable software components (RSC),” Day adds. “If you are doing an EO-178 certification, you need to certify the application and the OS, even If the RTOS can provide help. If you take an RTOS or other software component through certification, it will be certified by the FAA as an RSC. If you then provide that RSC certified RTOS to a customer building a safety-critical application, they don’t have to recertify the OS.”
Complex avionics software
This approach is significant because FAA officials see the complexity of software in avionics systems as so great that the cost of certification has become prohibitive. “So with RTOS at the heart of everything, the ability to have an RSC is a huge advantage that could save millions of dollars because they don’t have to go through all that line-by-line checking,” Day says.
“The other certification bodies are not as geared toward the software part and so may not go to the same depths in software certification as the FAA,” Day adds. “But anything certified to flight-critical by the FAA would give a medical system a higher level of confidence, even though the certification doesn’t translate one-to-one. But it does give the user some guidance on which OS may be the best starting point.
“I think those other certifications will become more software oriented as those applications become more networked and complex, which has long been the case with avionics. And that is where the advantage of being an FAA certified OS will really come through,” Day continues. “Both safety and security, although driven first by commercial avionics and military aerospace applications, soon filter down to adjacent industries.”
To keep up with the changes in play, RTOS vendors keep in close contact with hardware and software designers to determine where those capabilities are going and how they can interface with requirements they are hearing from their customers. As a result, variants of an RTOS may be introduced every year, with a full OS upgrade every three years or so.
Fully certified variants of an RTOS, such as the Green Hills Integrity, include those meeting industrial automation, FDA medical, and padded-cell technology standards. Padded cell also supports the operation of several operating systems on the same processor.
“All are safety standards and have some overlap, but are not identical; they all have different perspectives, so you need to redo the material and follow different procedures to meet the testing requirements for industrial automation or avionics or medical or padded cell,” Gicca notes.
“Aerospace and military heavily overlap in the sense that a lot of avionics applications apply to both, with the most heavily used RTOS tending to be in fly-by-wire flight control systems,” Gicca says. “In a push to use commercial products along with commercial processors, the military has chosen the FAA DO-178B safety critical standard for avionics, for which our Integrity 178B is certified. In other military applications, such as platform control systems, we use our general Integrity version; the costly rigorous test and design process for 178B isn’t required for a ground vehicle.”
Move to standard OS
John Wemekamp, chief technology officer at Curtiss-Wright Controls Embedded Computing in Leesburg, Va., says the majority of his company’s primarily military customer base has moved away from nonstandard operating systems to an almost exclusively embedded RTOS requirement in the past decade. On the commercial aerospace side, which he agrees has become a standard for military aerospace, the time-space partitioning of ARINC 653 has become the new standard, easing safety-critical software certification.
“The biggest growth is in the efficiency they (embedded RTOS) offer for developers, using IDE (Integrated Development Environment) to build more-complex systems and achieve a faster time to market, which everyone dealing with the military is under pressure to do,” he says. “Recently, everyone has focused on adding in memory protection; going to time-space partitioning made it easier to build more complex software environments.”
Embedded RTOS is not without limitations of its own, however, with users selecting the RTOS-as well as overlay operating systems and even languages-that provide the best SWAP and cost fit for their requirements.
“Runtime cost is probably the biggest thing that has been driving people to Linux, because of the open-source, free runtime world it offers,” Wemekamp says, adding another consideration is version control and the need to keep track of different options offered by different versions. “Longevity of support of older versions is another challenge, especially for the military, which has very long life cycles and doesn’t like to deal with constant new revisions.”
There are enabling technologies, in place and in development, that will address some of those concerns, as well as advance the overall capabilities of embedded applications and RTOS, he adds, including improvements in “user friendly” IDE.
“The other big thing is a move toward multicore processing-with the OS able to deal with that environment. The multicore challenge is really related to how to build parallel applications, which is probably the biggest challenge for the software environment. The software architecture needs to keep up, which is probably a longer challenge than that facing the hardware side,” Wemekamp says.
Code for new chips
New chips such as the Sun Microsystems UltraSPARC4+, IBM’s Cell, and other multicore solutions are of special interest to users looking at embedded solutions for complex safety- and security-critical applications.
The 64-bit UltraSPARC4+, for example, is fifth generation and the second for dual core, offering mission-critical throughput, chip multithreaded technology and the latest 90 nanometer process technology. Cell-also known as the Cell Broadband Engine Architecture (CBEA)-was designed around the diverse requirements of cryptography, graphics transform and lighting, physics, fast-Fourier transforms (FFT), matrix operations, and scientific workloads.
Wemekamp says NASA and others with high critical requirements also are interested in another SPARK-the high-level programming language and toolset family designed for writing software for high-integrity applications from Praxis High Integrity Systems in Bath, England. SPARK is based on an Ada language subset, making all SPARK programs legal Ada programs.
Next-generation system-on-a-chip (SoC) developments and applications are expected to further accelerate the use of concurrent programming techniques. As the transition from single- to dual-core machines already has begun at the desktop-PC level, embedded applications using multicore technology and energetic concurrent programming are expected to become equally common in security- and safety-critical arenas.
“Cell is a little different from traditional multicore; its complexity probably doesn’t allow it to be easily ported by every RTOS vendor. Multicore is a lot of single processors, so the RTOS will just run multiple processors, which is an attraction of the traditional multicore over IBM’s fairly unique and complex Cell structure,” Wemekamp says. “SoC concepts are based on using standard core processors and the RTOS vendors look at those on a business-by-business case for support. If the core is a processor they have traditionally supported, they are more likely to support the SoC based on it.
“Performance is always an issue, especially in areas such as stacks versus TCP/IP. TOEs (TCP/IP Offload Engines) OS vendors are taking advantage of those features and now need to support them more. Another (enabler) is Eclipse, which is an open source IDE activity,” Wemekamp continues. “Some vendors are supporting that, which allows third parties to build additional tools for the embedded environment that tie into the standard vendor tools. That should help progress to the next step in terms of more collaboration in the IDE environment.”
Open standards
The increasing use of open standards-such as Linux and POSIX-for embedded applications is expected to play a major role in RTOS applications, initially in the military and aerospace markets, then filtering down to other industries.
“We’ve found that Linux, which is becoming an interesting disruptive force throughout the embedded world, fits very nicely on top of a POSIX API. So what we offer is not just a set of RTOS that meet safety and security applications, but one you can run Linux applications on top of,” Day says. “This is particularly relevant in the military arena, where the use of open standards and operating systems like Linux are making reuse and the programming of systems that much easier. There has been a lot of debate about using Linux in the military, but we believe that will happen, although it will never be in an FAA or MILS certification because it is too big. But for everything that is not high safety or high security, Linux can do a good job in soft real-time.”
While not as important to the RTOS community, largely because of the long gaps between significant upgrades, language also has an impact of what RTOS vendors do. Day says most LynuxWorks users are working in C or C++, but also must be ready to support those, especially legacy applications, written in Ada.
At AdaCore in New York, CEO Robert Dewar says he believes an Ada revival is coming in the form of Ada2005, which has been enhanced for embedded developers-about half AdaCore’s market. He identifies Wind River, Green Hills, and Linux as the fiercest of the embedded competitors, citing the battle between the first two for the new Boeing 787, which he describes as having about half Ada-based avionics.
“One trend we’ve seen in the last decade is a lot of these systems were done on bareboards with specialized runtime systems, but now are over standard executives such as VxWorks,” Dewar says. “That has been a real change in the scene in the embedded world,” he says. “Partly that is due to much bigger processors that can handle fair-sized executives, so we are not as resource-starved as we were a decade or two ago.
“We see the same thing in space applications,” he continues. “The first safety-critical application we were involved in was the Canadian space arm on the International Space Station, which is a bareboard application using Ada. Eurospace also is doing a lot of stuff in Ada; the Ariane 5 was all Ada code, although poorly implemented Ada code because they failed to test it first.”
Homeland security
Homeland-security requirements also raise a new level of concern for embedded applications, Dewar adds.
“Safety and security applications are where Ada is being very closely considered. DHS is pushing much more concern with cyber security and secure programming. It is one thing to write a program with no bugs, but an entirely different level of complexity to write a program no one can attack. Ada is very suitable in that area,” he says. “Critical embedded programs are being written with multiple tasks and threads, which is quite difficult; Ada has always had threaded concurrent programming as an absolutely central concept.
“We’re just getting into security applications, as is Ada,” Dewar continues. “The most significant penetration there comes from the use of SPARK because, in security applications, extensive testing is useless to show a program is impregnable to hacking. So if the security of the nation depends on a program that can’t be attacked, we don’t trust testing. The NSA (National Security Agency) is interested in mathematical formulae that will prove the safety of our programs. Praxis is well ahead in that.”
Dewar says the problem is twofold: certifying extremely complex chips and the software they will run.
“Cell computing introduces very significant additional software complexity and I don’t know if we know yet how to handle that well; hardware always runs ahead of software,” he says. “At one point, we were going to simpler programming and simpler chips, but the current incarnation of the MIPS chip has some 50 million transistors and our level of confidence is not as great as when it had 50,000. Some bugs that would be caught in software certification are not caught when put into hardware because we don’t know how to do that level of certification in hardware. Intel is getting very interested in formal math techniques to certify their chip designs, but a lot of that is very secretive.”
That also applies to certifying complex object-oriented programs, for which the FAA is currently writing a new certification paradigm.
“The problem is certification must know the flow of control very well so you can test all possible flows, where in object-oriented, the state of the object determines the flow,” Dewar explains. “Our approach to certification now is exhaustive testing of all possibilities, which is tough to do in an object-oriented environment. After 50 years, you’d think everything in software was understood and solved, but we’re far from it.”
Real-time software glossary
API - Application Programmable Interface
CBEA - Cell Broadband Engine Architecture
COIL - Core OS Interface Library
EAL6+ - Evaluation Assurance Level 6-augmented
FFT - fast-Fourier transform
IDE - Integrated Development Environment
MILS - Multiple Independent Levels of Security
NSA - National Security Agency
RSC - reusable software components
RTOS - real-time operating system
SoC - system-on-a-chip
SRTOS - Secure Real-Time Operating System
TOEs - TCP/IP Offload Engines
