Artificial intelligence (AI) in unmanned vehicles

Computer scientists and unmanned vehicles designers work to find the right balance between machine autonomy and human-assisted operations in the next generation of military systems on land, at sea, and in the air.

By J.R. Wilson

One of the biggest buzzwords in military and commercial technology development today is autonomy — self-driving cars; aircraft with no pilot onboard; land, sea, and underwater vehicles without human controllers.

Unmanned aerial vehicles (UAVs), remotely controlled by humans, have been a key part of the world’s militaries for 20 years, led by the U.S. and demonstrated with devastating effect during the Second Gulf War.

The first U.S. UAV, which at first used what is now considered a primitive technology in the First Gulf War, was merely following the path created by Israelis in the 1980s as they sought to reduce the number of human pilots killed or captured flying low altitude surveillance missions over hostile territory.

Yet unmanned flight is one thing; and autonomous flight can be something else. None of the thousands of UAVs produced and deployed by nearly every nation on Earth in the past 20 years has been fully autonomous; there always has been a human in the loop.

True autonomy would remove the human element, with the autonomous unmanned vehicle (AUV) operating entirely on its own, reporting back to humans on what it has found during an intelligence, surveillance and reconnaissance (ISR) mission. Potentially, such platforms also could be armed and act on their own in a hunter/killer mode, but current international agreements prohibit any armed attack without a human “pulling the trigger”.

Full autonomy requires the use of another major technology challenge — artificial intelligence (AI) — which is replete with its own controversies, not only technological, but also ethical, moral, and legal.

The Textron Systems Common Unmanned Surface Vehicle is the U.S. Navy’s first unmanned surface vehicle program of record.

History of autonomous vehicles

What may surprise many who believe all these to be wonders of the 21st century’s new age of technology is the relatively long history of unmanned and autonomous vehicles. This history far predates their introduction into the modern world in the late 20th Century.

What is generally considered to be the first autonomous vehicle, in fact, was designed by Leonardo da Vinci around 1500. His self-propelled cart used high-tension springs to move without being pushed or pulled and a steering mechanism that could be set in advance to send it along a predetermined path. As with many of da Vinci’s inventions — including the helicopter and parachute — there is no evidence his cart was ever built.

The first autonomous platform actually produced and deployed was the Whitehead Torpedo in 1868. Inventor Robert Whitehead took existing short-range torpedoes, which traveled under momentum from their initial firing, and installed a pressurized system that enabled them to propel themselves underwater, maintaining depth, for several hundred yards. It had the same impact on naval warfare at the time as the Predator UAV had on aerial warfare nearly a century and a half later. Torpedo guidance also evolved rapidly after that, with expanded influence on other weapons, including aircraft.

The autopilot, part of nearly every major aircraft in the world today, was invented in 1933 by Sperry Gyroscope Co. in New York and gained worldwide fame when pioneering pilot Wiley Post used it on a 13,000 mile, round-the-world flight. “Mechanical Mike” used gyroscopes to track the aircraft’s heading by interfacing with flight controls to keep it on course.

Another form of autonomy is the car cruise control, which maintains a set speed while engaged. Although the Teeter Cruise Control was invented in 1945, it did not see commercial use until 1958.

The race to the moon led to the first self-driving vehicle actually built, the Stanford Cart, in 1961, when engineering graduate student James Adams proposed a way for a lunar rover to move about the moon’s surface despite the 2.5 second transmission delay from remote controllers on Earth.

The Textron Systems Aerosonde HQ is a variant of the company’s proven Aerosonde Small Unmanned Aircraft System with vertical-take-off-and-landing capabilities.

In 1977, Japan-based Tsukuba Mechanical Engineering advanced the concept of self-driving cars with a passenger vehicle that used cameras to detect street markings while moving at nearly 20 miles per hour.

A decade later, German engineer Ernst Dickmanns took that a step further, placing a bank of cameras and 60 microprocessing modules on the front and back of a car and creating what he called “dynamic vision” — an imaging system that focused only on objects relevant to the vehicle’s movement. His VaMoRs eventually was able to travel at speeds up to 60 mph on Germany’s high-speed Autobahn.

From 2004 to 2013, the U.S. Defense Advanced Research Projects Agency (DARPA) in Arlington, Va., sponsored a series of challenges for academic, industrial, and private inventors to develop a car that could self-navigate through a variety of environments, from a 150-mile desert course in 2004 to a 60-mile urban setting in 2007. While none of the entrants completed the desert challenge, four cars succeeded in traversing the urban route within that challenge’s six-hour time limit.

AI and autonomy

Since then, self-driving trams, taxies, and personal vehicles have been introduced, with varying degrees of success. Accidents involving these vehicles have slowed what some had expected to be a rapid transition in urban transit, but continuing advances in technology make their eventual widespread adoption a virtual certainty, with Chinese computer scientist and AI expert Andrew Ng recently saying they “will join human drivers on our roads sooner than most people think”.

Ng also predicts the development of true AUVs will require a “public-private partnership and a community of legislators and researchers and technology companies and automobile manufacturers.”

There is considerable disagreement across industry on when fully autonomous systems will be deployed by the military. All agree such systems will require at least a low level of AI — and therein lies the most oft-cited problem: The legal, moral and ethical considerations involved in having a robotic platform capable of self-guidance and action on the battlefield — even more so if that platform is weaponized.

Scott Engle, director of business development and capture at Mercury Systems in Andover, Mass., addressed both of the overarching questions surrounding AI and full autonomy: How soon and how likely?

HR3000 is a rugged, half-rack width, short depth 3U server. Part of Systel’s Puma-Bolt space saver server series, which is being fielded for Marine expeditionary vehicle programs.

“There clearly is the ability technologically to do a lot of systems autonomously, but the industry realizes if there is a catastrophic accident, the public’s confidence will be diminished and their support lessened,” he says. “The industry is proceeding cautiously, but not at revolutionary breakneck speed, which I think is appropriate.”

Military leaders may be even more cautious when it comes to AI. “By their very nature, the military is less cautious,” Engle says. “The question about full autonomy on the battlefield is do you want a system that does not have a human in the loop controlling when weapons are released, so the military is very cautious about deploying fully autonomous systems with the ability to do harm. More quickly will be support systems, such as medevac and supply delivery.”

Sean Baity, Technology Director for Unmanned Systems at Textron Systems in Hunt Valley, Md., agrees about the speed with which such systems can be produced.

Rapidly evolving technology

“The technology is evolving very rapidly. The hardware and software aspects are fairly accessible, no longer requiring exquisite platforms or yet-to-be developed math; they are now point and click. Right now it is building confidence in those systems in more challenging environments in commercial and military applications,” he says.

“That is going very quickly, with sizable investments and defense interested in bringing that level of rigor and confidence into defense systems,” Baity says. “One area leading this effort is UUVs [unmanned underwater vehicles], which are out there now and can operate for months at a time without any human intervention. That is a challenging environment, but does not have the considerations of civil air space safety.”

AeroVironment Inc. in Monrovia, Calif., has a long history in the production of UAVs for the military and, more recently, for commercial use. Steven Gitlin, vice president of corporate strategy, also sees the technology outpacing the military’s concept of operations (CONOPS).

The Textron Systems Shadow Tactical Unmanned Aircraft System has more than 1 million flight hours of experience from operations around the world.

“It’s a quickly evolving space with lofty ambitions and a lot of horsepower and we’ve been able to deliver some basic autonomy capabilities that we believe, over time, can deliver more value to the customer,” Gitlin says. “Anything that can enable a customer to perform their mission more safely, efficiently and cost-effectively is attractive. And that’s what unmanned vehicles have done already. It’s a continuum.”

In terms of technologies necessary to enable the next generation, he adds, “there’s a lot of development going on in machine vision, AI, neural nets, machine-to-machine communication. Those calls need to happen very quickly and the more complex the environment and task, the faster those calls need to take place.” AI’s role will be making the right decisions at the right time to achieve the intended outcome, being able to recognize a situation and make a decision that reaches the appropriate outcome, he says.

One of the most enthusiastic supporters of AI-based systems is Lockheed Martin Corp. in Bethesda, Md. In a January 2019 paper, the aerospace giant predicted a future where AI plays a major role in just about every part of military and civilian life.

“In an AI-enhanced future, humans will become better at everything; they’ll also become safer and less vulnerable to danger,” the paper states. “AI-enabled autonomous systems are changing the way militaries operate and protect their forces, the way first responders fight fires, how researchers explore the far reaches of space and the ocean’s depths.”

“Effective manned/unmanned teaming reduces the high cognitive workload, allowing the warfighter to focus on creative and complex planning and management,” the paper continues “Autonomous systems also have the ability to access hazardous mission environments, react more quickly and provide persistent capabilities without fatigue.”

Some analysts predict global sales of $100 billion in unmanned vehicles through 2025, 80 percent of that on UAVs. But those figures do not account for AUVs, which, if shown to be successful, could raise the figure far higher for military and commercial sales combined, with one of the largest applications being agriculture, where automation already has taken a strong hold.

The Textron Systems Aerosonde Small Unmanned Aircraft System has more than 300,000 flight hours of experience from land and sea operations around the world. At any given point in time, there is an Aerosonde system operating somewhere in the world.

Autonomy in unmanned vehicles

Boeing, the world’s largest aerospace and defense company, also is a major supporter of unmanned technologies today and fully autonomous unmanned vehicles tomorrow. In mid-February, the U.S. Navy awarded Boeing Defense, Space & Security a $43 million order to build four Orca Extra-Large Unmanned Undersea Vehicles (XLUUVs). At seven feet in width, the XLUUV is among the largest unmanned submersibles ever conceived for long-endurance surveillance missions or undersea cargo vessels to deliver sensor payloads and other UUVs.

The Navy and DARPA have been working with Boeing and Lockheed Martin for several years on a variety of large UUV efforts, such as the Large-Displacement Unmanned Underwater Vehicle project. The LDUUV is an autonomous submarine no larger than 80 inches in diameter that will be used to help develop concepts for using the larger XLUUVs.

While the operating environment for UUVs is less complex than that for either UAVs or UGVs, it nonetheless presents several unique obstacles for an unmanned system. First, direct communication from a manned submarine or surface ship is impossible. Instead, the UUV must cope with rapidly changing thermal layers to large aquatic animals like whales, to fishing nets, minefields, and even sunken ships. To be successful, a long-range underwater vessel needs some form of AI to handle those obstacles as well as changing conditions related to its mission.

AI is actually a multi-layered concept, ranging from machine learning, to adaptive reasoning, cognitive computing, and to full artificial intelligence. The first has been in use for several years, the second more recently, the third is still under development, and the final step has some way to go. Each, however, has a role to play in AUV development and deployment.

“AI is a software function,” says Mercury’s Engle. “The key with software in general is programmers have sets of requirements they have to write software to answer. But with autonomy, you can’t possibly list every possible problem you may encounter while flying, for example. We can only program for what we know the problems are today. The promise of AI is the ability to create software that can handle a situation it hasn’t encountered before, approaching the problem much the way a human would, drawing on past experience to deal with a new problem.”

Technology eventually is expected to catch up. “Technology is evolving,” says John Bratton, Mercury’s director of product marketing for sensor and mission processing. “In terms of processing, now we have the ability to process a lot of the algorithms needed for AI. But an overarching theme has to be infrastructure. For example, the ability of an unmanned vehicle to have an understanding outside the norm will depend on the increased bandwidth of 5G, which also is very low latency. And a lot of decisions will need to be made in real time.”

Autonomy and safety

One concern of those involved in the development of AI and AUVs in the U.S. is the possibility of falling behind two of the nation’s chief rivals and potential combat adversaries: China and Russia. Both have made it clear they are working hard on both technologies — and China has boasted of its advances in 5G. In addition, neither government faces the potential weight of public opposition should something go wrong.

“In the last 10-to-15 years, across the world, we’ve gotten used to the concept of digital convergence,” Bratton says. “The functionality that goes into smart phones, with a full back-up of cloud data center resources behind it, it will be a critical enabler for autonomy. Those same technologies are now being seen in other applications, such as automotive.”

Falcon-Strike is a C5ISR small form factor, high performance multi-mission computer fielded for fixed and rotary wing missions as well as ground-vehicle counter-UAV applications.

Safety also plays a big role. “There is the potential for us to be behind the Russians and Chinese [in military AI and AUVs], who may not have the same safety mindset,” Bratton says. “The U.S. has a goal of saving every possible person and minimizing the risk to every soldier. But that’s more sociological than technical. This is a brand-new domain and we don’t fully understand the liability issues. If an autonomous vehicle runs someone down, who is liable — and what were the circumstances involved? There are no absolutes with AI outputs, so it will be a little more interesting.”

The extensive certification programs are almost certain to change significantly.

“Going through certificating with an air vehicle currently is very long and arduous, with only a few companies seeking certification,” says Mercury’s Engle. “But there are hundreds of startups popping up all over the world that eventually will show up at the FAA [and its overseas equivalents] and ask to be certified. How to scale this industry to handle the evaluation and approval of all those systems is something we have to resolve. And that’s just aviation. Before a ground vehicle can be deployed, does some federal agency have to validate it is ready to go? I think that is another thing slowing this industry down.”

Textron’s Baity says he believes the true force driving future AUV development will not be the military, but the commercial world.

The role of commercial AI

“The economy of scale for the components that enable progressive autonomy will be important, such as the computing and sensor and data fusion capabilities developed for the auto industry,” Baity says. “That also impacts perception. Fully autonomous systems will emerge in areas with a low probability of causing harm, such as underwater or in space. We will see them in commercial applications such as warehouses, but basically an environment that can be described in a deterministic way. To avoid safety issues or overburden the military user is not useful.”

Much of it depends on the evolution of computer technology, Baity continues. “The computation aspects will continue to progress and increase capabilities, especially in the perception aspect, in resilience and modality, not depending on one form of perception but a combination of sensors. For the foreseeable future, humans will remain on the loop as we address the cognitive workload of that operator to manage that system, which involves systems that can provide the right answer to manage those systems effectively. The other aspect is verification. In defense, you have programs that run through proving out and developing confidence, which will remain, but the economics of that are high. Standards compliancy and plug-and-play are part of that to create technologies we can afford to deploy.”

FLIR Systems in Wilsonville, Ore., acquired Endeavor Robotics (nee iRobot) last February, is expanding its portfolio from sensors to robotic systems, with a long-term focus on autonomy.

“In the future, we will be looking to integrate manned and unmanned systems across multiple domains, including unmanned air and ground systems, collaborating in the battle space as well as working with manned systems,” says Roger Wells, vice president/general manager for unmanned systems & integrated solutions at FLIR’s Government & Defense Business Unit. “This gives us the opportunity to move from sensors, where we have a long and robust history, to intelligent sensors and sensing to fully integrated solutions designed to meet the needs of the missions.”

It’s likely we will see more of this in the future. “We see unmanned autonomous systems becoming much more pervasive, in our personal lives and on the battlefield,” Wells continues. “It is evolving by leaps and bounds every day. We are getting to the point in our technology development — especially in terms of SWaP [size, weight and power] — where we can package tremendous amounts of capability into small units. The architectures are allowing the systems to more effectively integrate into the force structure as well as operate in autonomous ways across a wide spectrum of missions. The analytics, AI and applications are allowing our systems to be more effective at understanding the environment and conducting their mission relative to what they are sensing. This is providing a unique set of capabilities to small units on the front line, with systems around them supporting that.”

Enabling technologies for AI

To handle a fully autonomous system, from navigation to sensors to communications — perhaps, eventually, weapons — will require a complex, multi-level AI system with a single interface to all of those to avoid SWaP parameters that make the system too large to be practical.

“Through integrating, ruggedizing the right hardware, we can enable an AI footprint in a single system rather than several difference pieces in the past,” says Aneesh Kothari, vice president of marketing at rugged computer specialist Systel Inc. in Sugar Land, Texas. “It also provides a single point of integration for all the sensors these platforms will carry. The number of things now possible is quite remarkable. And such systems divert weight and operational requirements from the individual soldier.”

Many improvements have come only recently. “It seems to be evolving very rapidly, especially in the past two years,” Kothari says. “You can see that across the military services, such as the Army’s robotic combat vehicle, smaller-form-factor UAVs, UGVs being looked at as forward deployed units, the Navy’s autonomous helicopter. The mandate for AUVs is quite common across the board, making what was science fiction only a few years ago quickly becoming reality.”

Unmanned systems have proliferated across all domains, from the Boeing-built U.S. Air Force unmanned X-37B mini-shuttle, to the Orca, to driverless cars to UAVs. The next step, fully autonomous platforms, appears to be technologically close, and depends only on advances in artificial intelligence.

“AI is a vision of what could be, with a tremendous amount of work underway, such as machine learning and other aspects that simplify things down to a ‘check engine’ light level,” says Textron’s Baity. AI will make sure that confidence is established and maintained in the system and mitigates the cost of training and maintenance. There is a desire to minimize the amount of training required so you can walk up to a system and know how to use it, much as you do with a telephone.

“The area of autonomy is a broad one and people feel very passionately about it,” Baity continues. “It is something that will benefit our lives and our ability to get the job done. We want to make sure they are extensions of human intent and managed in that manner, covering customer needs in air, land and sea.”