DARPA eyes artificial intelligence (AI) to embedded computing in high-end video processing at the edge

ARLINGTON, Va. – U.S. military researchers are trying to reduce the complexity and enhance the efficiency of artificial intelligence (AI) video processing to enable the use of high-end vision sensing with embedded computing hardware at the leading edge of the battlefield.

Experts aim to do this by designing front-end AI algorithms into imaging sensor pixels to reduce the high data streams typical in conventional image processing by at least 10 times.

Officials of the U.S. Defense Advanced Research Projects Agency (DARPA) in Arlington, Va., issued a solicitation last Wednesday (DARPA-PA-20-02-09) for the In-Pixel Intelligent Processing (IP2) project.

The solicitation, called an artificial intelligence exploration opportunity, focuses on low-power data extraction at the tactical edge.

Related: The ever-shrinking world of small-form-factor embedded computing

The IP2 video processing project seeks to reclaim the accuracy and functionality of deep neural networks in power-constrained sensing by developing AI algorithms matched to in-pixel mesh processing layers to bring the front end of the neural network into the sensor pixels and inject intelligence into the data stream at the sensor edge.

IP2 will enable efficient and embedded 3rd-wave AI for object detection and tracking in large-format high-frame-rate vision sensors. New front-end AI algorithms designed into the sensor pixels will learn saliency through the statistics in the data to create a reduced data-stream that enables 10x more efficient compact AI processing.

The IP2 program seeks to bring the front end of the neural network into the pixel, to identify salient information, and simplify video data at the source. Recent simulations have shown it may be possible to create in-pixel neural network structures that identify salient information and reduce the data complexity at low power and within the footprint of an individual pixel.

IP2 also will implement closed-loop task-oriented algorithms on embedded computing hardware like field-programmable gate arrays (FPGAs). Recent evidence from a DARPA Joint University Microelectronics Program (JUMP) exploration suggests that creating structure in the video data stream can enable hardware specialization, higher efficiency, and new functionality for processing 3rd-wave AI perception models.

Related: Artificial intelligence and machine learning for unmanned vehicles

The first phase of the IP2 project will develop front-end AI task-oriented saliency and dimensionality reduction algorithms and train neural network architecture on full HD frames showing a 10x reduction in dimensionality and data bandwidth; and design in-pixel neural network circuits able to run these algorithms.

The project's second phase will design an in-pixel mesh hardware emulation; demonstrate emulated 250 milliwatt per megapixel for a full-format sensor with programmable in- pixel circuitry; develop a test bed; and make plans to fabricate the full-format sensor.

Companies interested should upload proposals no later than 3 June 2021 to the DARPA BAA website at https://baa.darpa.mil. Email questions or concerns to Whitney Mason, the IP2 program manager, at [email protected].

More information is online at https://beta.sam.gov/opp/64ae26eed1f446588bf2bdc5e1e2c936/view.