Military seeks to train and aid medics and mechanics using artificial intelligence (AI) and augmented reality
ARLINGTON, Va. – U.S. military researchers are asking industry to develop artificial intelligence (AI) technology that perceives the environment, reasons about physical tasks, and models the user -- all in real-time.
Officials of the U.S. Defense Advanced Research Projects Agency (DARPA) in Arlington, Va., issued a broad agency announcement (HR001121S0015) last Friday for the Perceptually Enabled Task Guidance (PTG) project.
The goal is to enable military mechanics, medics, and other specialists to perform tasks within and beyond their skills by providing just-in-time feedback and instructions for physical tasks.
The AI will capitalize on commercially available sensor technologies to see what the user sees and hear what the user hears, and will provide information and instruction to the user using augmented reality.
Related: The future of artificial intelligence and quantum computing
Military personnel must perform an increasing number of tasks and more complex tasks than ever before, DARPA researchers explain. Mechanics, for example, must repair more types of sophisticated machines and platforms, and Medics must perform more procedures over extended periods of time.
The goal of the PTG program is to make users more versatile by expanding their skills and more proficient by reducing their errors by using AI assistants that provide just-in-time visual and audio feedback.
Users of PTG assistants will wear head-mounted cameras and microphones so that the assistant can see and hear what the user does. Users also will wear augmented reality headsets that enable assistants to provide auditory and graphics feedback.
The assistant will learn about the user's tasks by ingesting knowledge from checklists, illustrated manuals, training videos, and related information.
Users may ask the assistant questions like “what do I do next?” When a user makes a critical mistake, the system should warn the user and suggest remedial action. When a task is new to a user, the assistant should walk the user through the necessary steps.
PTG technology will take advantage of recent advances in deep learning for video and speech analysis, automated reasoning for task and plan monitoring, and augmented reality for human-computer interfaces.
The program, moreover, is looking for ways to address four key problems: knowledge transfer, perceptual ground, perceptual attention, and user modeling.
Knowledge transfer will enable AI assistants to learn from instructions intended for humans, with an emphasis on checklists, illustrated manuals, and training videos.
Related: EO/IR sensors boost situational awareness
Perceptual grounding will align the objects, settings, actions, sounds, and words recognized by an assistant with the terms it uses to describe and model tasks, so as to map observations to its task knowledge.
Perceptual attention will enable assistants to pay attention to what's relevant to tasks, while ignoring extraneous stimuli, and respond to unexpected events that may alter a user’s goals or suggest a new task.
User modeling, meanwhile, will help the AI assistant determine how much information to present to users and when to do so, taking into account what the users, a physical model of what the users are doing, and a model of how well the user can pay attention based on his emotional state.
Related: The emerging world of hypersonic weapons technology
The PTG program is seeking proposals in two technical areas -- the first for fundamental research into knowledge transfer, perceptual grounding, perceptual attention, and user modeling; and the second for integrated demonstrations of assistants for mechanics, medics, or aircraft pilots.
DARPA experts will brief industry virtually from 1 to 4:45 p.m. on 18 March 2021 via Zoom Webinar. Register for the briefings online at https://www.schafertmd.com/DARPA/I2O/PTG/PD/?p=registration. More information about the PTG proposers day briefings is online at https://www.schafertmd.com/DARPA/I2O/PTG/PD/.
Companies interested should submit abstracts no later than 31 March 2021, and full proposals no later than 14 May 2021, to the DARPA BAA Website at https://baa.darpa.mil. More information is online at https://beta.sam.gov/opp/39d76d1a5c684eb0a6c834f17aaa678d/view.
John Keller | Editor-in-Chief
John Keller is the Editor-in-Chief, Military & Aerospace Electronics Magazine--provides extensive coverage and analysis of enabling electronics and optoelectronic technologies in military, space and commercial aviation applications. John has been a member of the Military & Aerospace Electronics staff since 1989 and chief editor since 1995.
