Next-generation persistent ISR will require more signal processing and data fusion on the sensor platform, as well as precision approaching artificial intelligence.
BY J.R. Wilson
Global persistent surveillance conjures images of innumerable National Security Agency (NSA) analysts poring over sensor data covering every square inch of the Earth, 24/7/365. While there is a limited capability for gathering such data, using geosynchronous satellites carrying powerful sensors, there are not enough memory cores, bandwidth, human analysts, or real artificial intelligence products for such an approach to be practical.
This raises the first and - in the minds of many of those involved - perhaps the most important question: What is global persistent surveillance?
There have been several definitions offered, most similar to the Joint Warfighting Center's Joint Doctrine Support Division, which in 2011 issued a Commander's Handbook for Persistent Surveillance citing the joint doctrine definition as "a collection strategy that emphasizes the ability of some collection systems to linger on demand in an area to detect, locate, characterize, identify, track, target, and possibly provide battle damage assessment and retargeting in near- or real time. Persistent surveillance facilitates the prediction of an adversary's behavior and the formulation and execution of preemptive activities to deter or forestall anticipated adversary courses of action. An effectively executed persistent surveillance strategy greatly enhances joint military operations."
Today's persistent intelligence, surveillance, and reconnaissance (ISR) overlays all five military domains - air, land, sea, space, and cyber - not as individual activities by separate services, but as joint networked systems using real-time and archival data fusion.
With ever-increasing volumes of data gathered by more and better sensors in an already bandwidth-constrained battlespace, next-generation persistent ISR will require more signal processing and data fusion be done on the sensor platform, but with a precision approaching artificial intelligence.
To expand those definitions to "global" status currently requires a modification of the concept to "globally available persistent surveillance, on demand, whenever, wherever, and for any duration required by the end user." The request may come from a platoon sergeant who needs to know what is over the next hill, in real-time or a four-star geographic combatant commander needing an up-to-date overview of an entire country or region, up to the White House situation room, where the President, Secretary of Defense, and Joint Chiefs of Staff need both as wide a view of an area of concern as possible and a real-time look at who is doing what and where.
Given current technology and other factors like acquisition procedures and inter-service and cross-agency cooperation, some say this definition today is unrealistic.
Setting the bar high
"So far, that doesn't really exist, although the trends are moving in the right direction to enable it," says Steven Gitlin, vice president of corporate strategy at unmanned aircraft designer AeroVironment Inc. in Monrovia, Calif. "The question is, who is willing to pay for it at what level of persistence. We could deliver a similar capability today, if the customer is willing to pay for it, but it would cost and would have to solve a specific need."
Technologies involving solar power and batteries are coming to bear on prospects for global persistent surveillance, Gitlin says. "Advances in renewable energy, such as solar power, also are benefitting that kind of architecture. So as solar panel efficiencies have increased and costs have come down, the modes of integrating those into aircraft have increased. It's also new integration technologies on the platform and advances in battery technology, driven by the electric car and cell phone industries, which bode well for cycle time and other factors that will be important for high-altitude, long-endurance platforms."
On nearly every aspect of this level of persistent surveillance - from the nature of the target, terrain, weather, and time of day to who is seeking the information and for what purpose - the answer is "it depends" when considering what platforms and sensors are required, the level of detail sought, and even the definition of "real time."
"Looking at the mission, real time is whatever is required to give the decision-maker enough time and information to make the best decision, based on refined information and intelligence. We are definitely targeting our technologies in automated analytics and fusion because we see machines will be required to keep up," says Eric Vogel, program director at BAE Systems ISR Solutions in Nashua, N.H.
"The key challenge is keeping pace and being agile in a rapidly evolving environment, the ability to put what I need when and where I need it, even if I can't predict that in advance as well as I used to. All the sensing modalities are important, with cross-domain being very important for persistent surveillance," Vogel says.
Another change, which has been in progress for the past few years and now is gaining momentum, is making such systems software-defined - that is, changing various aspects of the system to fit changing mission and environmental requirements using software modifications rather than swapping out fixed-use hardware.
"Sensors that are software-defined and can be responsive to the environment and operations they are put in will be key technologies to fulfill those missions going forward. It needs to be flexible during the mission and over the life cycle to keep pace with a very evolving threat environment," Vogel continues.
Tailoring to the environment
"Scalable open-architecture systems leveraged from a core common architecture, in both hardware and software, enable you to tailor the sensors to match the platform environment and SWaP-C [size, weight, and power, plus cost] requirements, but you don't have to start from scratch each time. That's another way of being agile because it reduces NRE [Non-Recurring Engineering], cycle time, deployment time - persistent surveillance is all about a proliferation of sensors and this enables that."
While today's persistent-surveillance might not be close to the military's dreams of perfect capability, there still is plenty of embedded computing technology available that offers to move capabilities forward, says Marc Couture, senior product manager-digital signal processing and high performance embedded computing at the Curtiss-Wright Corp. Defense Solutions Division in Ashburn, Va.
"It's basically data that's actionable. There are cases where if it takes 10 minutes to identify a convoy, that might still be considered real time. Others measure it in milliseconds or even microseconds. A platoon doesn't want to wait for information, so real-time is mandatory. There is still a need for post-mission analysis, but a lot of what used to be discerned post-mission is more and more being done in real time," Couture says.
"A lot of what this technology is all about is doing what analysts have done for a long time, but more accurately and orders of magnitude faster. For example, Nvidia GPUs [graphics processing units] are trained to look for specific targets based on existing databases, learning while in-theater. With a limited data pipe, the higher the quality of data and more quickly it can be obtained, the better. AI [artificial intelligence] is going to impact all aspects of defense computing, but surveillance is where it is already starting to hit."
Artificial intelligence has a multitude of definitions, influenced by the state-of-the-art in computer technology, the needs of the user community, and the amount and speed of processing. Still, there remains significant concern about the roles of human processing and machine processing in the ISR environment.
"Agile ISR is a centerpiece, a key enabler for future operations. I've been in this business 20 years and have seen it evolve from stove-piped, hardware-driven solutions to COTS-based, software-defined solutions running with open interface standards and business models that drive innovation at multiple levels of the system," notes BAE Systems' Vogel.
"As sensor providers, we are looking at technologies that enable our customers to digest and quickly generate actionable intel from the wealth of raw data produced by these modern sensors. Advanced analytics and fusion processing are at the top of the list to help solve that problem, automating as much as possible. If you rely on humans to do it all, you won't be able to keep up with the wealth of data coming in or what needs to be disseminated to the right people at the right time."
Speed of technology
Current and future requirements for persistent surveillance have grown significantly because of major advances elsewhere in military technology, says retired U.S. Air Force Lt. Gen. David A. Deptula, dean of the Mitchell Institute for Aerospace Studies in Arlington, Va.
"As we have achieved greater abilities in accuracy and the delivery of effects, that has driven the need for persistent surveillance," continues Deptula, who from 2006 until his retirement in 2010 was the Air Force's first deputy chief of staff for ISR. "The side that wins in the future is the side that has the greatest situational awareness and acts most quickly."
The speed of decision making, not precision targeting, today is among the most important military capabilities. "The ability to hit a target is no longer an issue - the U.S. can strike any target anywhere on the Earth, all day and night, any weather, rapidly and with precision," Deptula says. "The issue now is what effect to you want to achieve, which is not always kinetic. The challenge is not lethal force, but to define and fix the target. In World War II it took a thousand aircraft dropping 10,000 bombs to destroy a target that today one aircraft can accomplish. The need for accurate information drives persistent surveillance."
What sensor technologies and platforms are necessary, and how they will be used remains a matter of debate - especially as rapidly evolving technologies influence those requirements.
Visible-light and infrared sensors today often work together with synthetic aperture radar, RF emitter data, and other relevant information. "A lot of the sensor fusion challenges involve overlaying all of those sensor types to get better overall data," says Curtiss-Wright's Couture. "Persistent surveillance is just a waterfall of data coming out of sensors with finer resolution - and we're now finally seeing sensor fusion, which has been talked about for decades. And processing systems that were too big and heavy 10 years ago are now possible."
The sheer volume of surveillance data available today is one of the biggest challenges facing systems designers. "The resolution per unit of surface area is much higher today; megapixel cameras are now gigapixel, SAR images have higher resolution, there are big advances in SIGINT, with far more accuracy than just a decade ago," Couture continues. "All the advancements in those technologies give you greater accuracy, but also mean you have a lot more data coming out of the sensors, so it's a good thing computing capability is keeping up with that deluge. AI at the cognitive level is coming out of its infancy, but still has a lot to be done in the next decade."
These new approaches will require vast use of new technologies, yet legacy systems also will play a role in future persistent ISR efforts.
Legacy systems "were designed to collect a specific type of data for a specific user. As we look at the speed of the threat and how it has evolved, we now realize there is better intel when looking across the entire set of data for context rather than looking at any one type of data for overall situational awareness and context," says Jane Chappell, vice president for global intelligence solutions (GIS) at the Raytheon Co. Intelligence, Information & Services segment in Dulles, Va.
"Any one piece of data by itself is interesting, but putting it in context with the overall environment brings a whole new level of intelligence," Chappell says. "Persistent surveillance is extremely important if you look at how our adversaries have changed their means and methods since the Cold War. That pace has changed in how they approach issues, how they fight a war, how they disrupt our capabilities, which makes it key for us to predict and automate and move at a faster pace than any adversary."
Human limitations
There also is a human limitation to persistent surveillance, especially as the volume of data keeps growing. This is one of the chief challenges of the U.S. Army Program Executive Office for Intelligence, Electronic Warfare & Sensors (PEO-IEWS).
"Apart from fusing data to a single operating environment, which is not a trivial issue, the persistent sensing of one area leads to analyst fatigue - looking at one environment for extended periods of time for what can sometimes be vague details," Mark Kitz, chief engineer at PEO-IEWS at Aberdeen Proving Ground, Md. "We mitigate that through algorithms for change detection and orienting the analyst toward other things to keep them engaged, but, in the end, they do get fatigued quite quickly, no matter the sensor technology."
One challenge is to balance the amount of sensor processing that humans and machines perform so that human analysts can stay focused. "We are trying to keep the analyst engaged in the sensor's area of activity so they can make decisions in a timely manner, maintaining that diligence and balancing how much is done using algorithms and how much is done by the analyst," Kitz says. "The focus of these algorithms, which I would not go so far as to call AI, is to ensure the analyst doesn't miss something. The key is to supplement, not supplant, the analyst."
Although the U.S. military today has a commanding lead in persistent surveillance technologies, other nations are moving from near-peer to parity. "Other nation-states are just as sophisticated as we are in some areas," points out Curtiss-Wright's Couture. "Cognitive EW, for example, is a cat-and-mouse game. Deep learning and machine learning tie into these cognitive efforts, not just EW but other areas, as well, where you try to get your adversary's sensors to pick up false information. Just the sheer number of assets the U.S. has kept us in the lead for now, but I feel we are in danger of falling behind in some areas; we should never get complacent."
There are several reasons that the U.S. technological lead in persistent surveillance may be diminishing - some of which are self-inflicted, says the Mitchell Institute's Deptula. "We've seen the globalization of technology and we have created fiscal limits on what goes into defense that are not based on national security. Meanwhile, our adversaries, because technology is becoming a commodity and they have been sending students to our schools, are closing the margin of advantage the U.S. used to have. And now it is getting to parity, which will be a challenge for us as we move forward."
Integrated base defense
While global persistent surveillance or something close to it may be the ultimate goal of major intelligence-gathering organizations, the most pressing need emphasizes the "globally available" concept. That need, providing constant ISR to defend areas of interest like forward operating bases, is a goal of the Army PEO-IEWS.
"Apart from anomalous ID detection, explosive detection, identification of threats, the technology we're most interested in for integrated base defense is standoff detection, so we can identify a potential threat prior to engagement," Kitz says. "We're highly invested in video detection capability and the algorithms of change detection and anomalous behavior, challenging industry on how to get more stand-off."
Even such a localized effort is not divorced from a larger ISR capability.
"Aerostats are the primary platform, although we also deliver towers and use UAVs [unmanned aerial vehicles]," Kitz says. "These sensors and platforms are not entirely dedicated to base defense, but are integrated into the force protection system so we have one picture of the area rather than just base protection."
One goal is to put persistent surveillance information in the proper context. "We interface with Army intelligence processing and overall military intelligence so we understand what is happening in the area and the operational environment in context of the security of the base. Those interfaces are critical to providing the base commander with an understanding of what is happening around the base."
As the area grows over which persistent surveillance is necessary, so does the need for consolidating all sources of information into one comprehensive picture. That capability, at least from a source perspective, has grown substantially since the turn of the century.
"The number of sensors and systems out there today is much larger, especially if you bring in commercial data sources," says Bob Canty, vice president for business development at Raytheon Intelligence, Information & Services. "Sharing data across all those systems is another enabling factor in achieving global persistent surveillance. That is where technology really comes into play-the reduction of sensor cost. But if you are bringing in open-source data, how do you track its pedigree throughout the effort so you have confidence in the end result?"
Using computers to put all that data together to free human analysts to spend more time analyzing is another goal.
"Visualization of data has always been incredibly important - and you want enough data to understand context and make decisions on consequences. We have spent a lot of R&D on understanding the target of interest and defining it in a way that the data finds the user rather than the user trying to find the data. In the past, roughly 80 percent of an analyst's time was spent looking for data. We want to reverse that, so they spend 20 percent looking and 80 percent analyzing," says Raytheon's Chappell.
"That's all about understanding what the target and area of interest are and making sure, through the analytics, that the data are searched in a way that it is reduced to a manageable set," Chappell says. "So, it is all about the analytics and automation - the speed and how we get the right data to the users for better decisions and how you use the results of that and move that data through the system to take the next set of actions."
While providing persistent surveillance for a small area, such as a forward operating base, might seem a lot simpler than doing the same across a wide geographic region, the Army's Kitz says that is not necessarily the case.
"Fifteen years ago, you were looking at a small area. Now, using hyperspectral capabilities, we can see a wider view and much further out to identify a potential threat or anomalous behavior, which make our demands for processing and on the analysts much higher," Kitz explains.
"Persistent surveillance requires infrastructure for a forward operating base - concrete, fiber, an aerostat, and a large ground station. The only other consideration is the investment that enables the integration of all these sensors, a contingency operations cell, analysts, etc. And the more sensors and data you have, typically the more infrastructure you require."
Rewinding the data
There also is a forensic aspect to persistent surveillance - the ability to rewind the data to track movements and developments leading up to an incident.
"Forensics can be enormously important when you are trying to trace the cause of a particular activity. At the same time, if I'm engaged in a real-time activity, I'm less concerned with forensics than I am with real-time knowledge of what my adversary is doing and how can I stay a step ahead," says the Mitchell Institute's Deptula. "The real-time aspect gives the edge over an adversary by building situational awareness, but that is not to reduce the importance of forensics, which can help develop options to greater strategic challenges."
BAE Systems' Vogel agrees: "Being agile and allowing customers to do either real-time or forensic analysis with a common sensing solution is the most important thing."
Software-defined systems will satisfy the bulk of requirements going forward, while advances in commercial technologies will enable those systems to have the performance required and the flexibility necessary to adapt to military targets.
"There are domains where MILSPEC sensors are still required, but even there you are seeing much more COTS influence than in the past. The whole chain has to be optimized to achieve an advantage," Vogel says. "If you have the best sensors, but the next step in the cycle isn't equally the best, then that's the limiting factor in the product you're generating. It's really a system-of-systems approach, from the platform through the sensor to the back-end processing, all of which are critical."
The ISR target environment is evolving more rapidly than it ever has. "Traditional technologies and acquisition models need to be updated to keep pace with that, Vogel says. "We're addressing that in two ways, from a technology standpoint - software-defined, open architecture sensors built on standards-based designs and open to technology insertion over their life cycles - and, of equal importance, open business models, giving customers the right positions and data so they can promote competition and innovation to get discriminating technology insertion when its needed."
As to the future, Raytheon's Chappell offers a more definitive look at real-time delivery. "As we continue to evolve our capabilities, it all goes back to the changing nature of the threat and our need to stay ahead of that threat. We have a lot of capability from a sensor standpoint; now it is all about networking those capabilities together, sharing the information across the different organizations at the speed and skill we need to understand patterns of life. And that means a big focus on automation and analytics," she says. "So, the key is delivering at the speed of need."
