DARPA seeks to develop real-time intelligence processor to uncover patterns in vast data
ARLINGTON, Va., 4 Aug. 2016. U.S. military researchers are asking for industry's help in developing a new data processor to help intelligence analysts understand relationships in vast data streams from cameras, social media, sensor feeds, and scientific data.
Officials of the U.S. Defense Advanced Research Projects Agency (DARPA) in Arlington, Va., issued a presolicitation Tuesday (DARPA-BAA-16-52 ) for the Hierarchical Identify Verify Exploit (HIVE) project.
The HIVE program seeks to develop a generic and scalable graph processor that specializes in processing sparse graph primitives, and achieves 1000-times improvement in processing efficiency over standard processors.
This capability will help intelligence analysts discover the relationships between events as they unfold in the field, rather than relying on forensic analysis in data centers, DARPA officials say.
The program will develop chip prototypes and software tools to support programming the new hardware, as well as design a system architecture to support efficient multi-node scaling.
Related: DARPA eyes sensors, communications, and data processing technology for the infantry squad
Today large amounts of data come from sources like social media, cameras, other kinds of sensor feeds, and scientific data. Graph analytics has emerged as a way to understand the relationships between these heterogeneous types of data to enable analysts to draw conclusions from the patterns in the data and to answer previously unthinkable questions, DARPA experts explain.
Yet analysts might be able to understand a more complete picture of the problem by understanding the complex relationships between different data feeds.
Today most graph analytics happens in large data centers on large cached or static data sets. This requires massive amounts of processing power -- particularly for “needle-in-the-haystack” types of problems. Moreover, the nature of the graph can be very sparse, as analysts are not clear on the number of relationships between entities.
Analysts also need to make decisions in real time. To do this, they must understand how relationships in the graph evolve over time. The graph must update at the speed of incoming data, not as an offline process, because the graph must develop and change in real time.
Yet trying to analyze the graph with standard processors is extremely inefficient because sparse data must be processed in real time, DARPA officials say. Graph analytics shifts the processing workload to locating the information and moving the data; only 4 percent of processing time and power goes to the overall effort.
Related: DARPA launches WIRED program to develop affordable wafer-scale infrared sensors
Such inefficiency either limits the size of the graph to what the chip can hold, or requires an extremely large cluster of computers. Instead, the HIVE program seeks to create a special processor that works on graph analytics 1000 times faster than can today's standard processors.
The program will focus on improving the efficiency of random access memory transactions to limit data movement, efficient parallelism to improve scalability, and new accelerators designed specifically for graph computation.
The HIVE program has three phases. First, an architectural phase will develop new memory controllers, new accelerators based on graph primitives, new data flow models, new data mapping tools, and new middleware to enable seamless transition of existing graph algorithms onto the new hardware.
Second, a prototyping phase will demonstrate these new technologies on military applications. Third, a fabrication phase will demonstrate the scalable performance of a 16-node system of custom graph processors for accelerating the most demanding military analytics applications.
The project has three technical areas: graph analytics processor, graph analytics toolkits, and system evaluator. The graph analytics processor -- technical area one -- will design a new chip architecture from scratch, focusing on the memory wall and on parallelization of multinode systems. The memory wall will create new memory architectures that allow for non-uniform memory access (NUMA).
Related: DARPA approaches industry for new battlefield network algorithms and network protocols
Parallelization, meanwhile, will enable machines to work closely together, rather than working in parallel but running independently. Technical area one, in essence, has to move from today’s single instruction multi-data (SIMD) world to one that allows for multiple instruction multi-data (MIMD) execution, DARPA officials say.
This component of the HIVE program will create an accelerator architecture and processor pipeline that processes graph primitives in a native sparse matrix format; develop a chip architecture that moves data quickly and efficiently from memory or I/Os to the accelerators; and develop an external memory controller that uses data mapping tools to handle random and sequential memory accesses on memory transfers as small as 8 to 32 bytes.
The prototype phase will develop the new chip architecture on a printed circuit board and low-level software that will emulate the future chip. Demonstration fabricate, test, and bring-up the new graph processing chip.
Graph analytics toolkits -- technical area two -- aims to develop the fundamental software technologies to translate existing graph algorithms into the new hardware by developing micro-code to match the microarchitecture of the new chips. Micro-code must support the data format and graph primitives of existing graph algorithms and not force them to re-write their algorithms.
System evaluator -- technical area three -- aims to identify and develop static and streaming graph analytics to solve five types of problem areas: anomaly detection, domain specific search, dependency mapping, N-x contingency analysis, and causal modeling of events.
Related: DARPA seeks human brain-like computers for complex signal processing and data analysis
Companies interested should submit proposals no later than 19 Oct. 2016 to the DARPA Website at https://baa.darpa.mil. Submit abstracts to the DARPA Website no later than 2 Sept. 2016.
DARPA officials will brief industry on the HIVE program from 9 a.m. to 12:30 p.m. on 11 Aug. 2016 at the George Mason University Arlington campus, 3351 North Fairfax Drive, in Arlington, Va. Register for the HIVE industry day no later than 9 Aug. 2016 online at www.cvent.com/d/vvqjz9.
Email questions or concerns to DARPA's Trung Tran, the HIVE program manager, at [email protected] .
More information is online at https://www.fbo.gov/spg/ODA/DARPA/CMO/DARPA-BAA-16-52/listing.html.
Learn more: search the Aerospace & Defense Buyer's Guide for companies, new products, press releases, and videos
John Keller | Editor
John Keller is editor-in-chief of Military & Aerospace Electronics magazine, which provides extensive coverage and analysis of enabling electronic and optoelectronic technologies in military, space, and commercial aviation applications. A member of the Military & Aerospace Electronics staff since the magazine's founding in 1989, Mr. Keller took over as chief editor in 1995.
