Neural computing research at Sandia covers the full spectrum from theoretical neuroscience to neural algorithm development to neuromorphic architectures and hardware. The neural computing effort is directed at impacting a number of real-world applications relevant to national security. These applications include improved cyber security and cyber defenses, embedded pattern recognition and data analytics systems, and neural-inspired approaches to improving scientific computing and high performance computing.
Neural Machine Learning
Neural-inspired machine learning algorithms such as deep neural networks and sparse coding systems have greatly impacted a number of data analytics domains. Our focus is on identifying paths to better leverage neuroscience knowledge in the development of novel and extension of existing neural machine learning techniques. Areas of research include:
- Enabling continuous learning in deep learning algorithms by incorporating neural processes such as adult neurogenesis (see Draelos et al., International Conference of Learning Representations).
- Developing models of content-addressible memory based on the hippocampus circuit architecture.
Formal Neural Computing Theory
While neural computing has often been identified as a post-von Neumann architecture option as computing moves into a post-Moore's Law era, the precise computational benefits offered by neural approaches have been challenging to pin down. Sandia is developing formal theoretical computing approaches to identify the aspects of neural computation that will yield the highest application benefit. These approaches have focused on three domains: the use of spike-based representations, the ability to colocalize processing and memory, and the incorporation of continuous learning and adaptivity in a comptuing system.
Neural Computing Architectures
The development of effective neural computing architectures requires not only the design of physical systems capable of simultaneously processing thousands or millions of neurons and synapses, but also the development of software interfaces and APIs to allow algorithm development and systematic validation and verificatino techniques to ensure the noise-tolerance and real-world validiy of these systems. Through the HAANA Grand Challenge, Sandia is vertically developing two different neuromorphic computing architectures.
- The spike-timing processing unit (STPU) is an event driven architecture leveraging existing CMOS technologies to rapidly process streaming data in the spiking domain.
- The resistive memory crossbar (ReRAM) is a crossbar based architecture leveraging novel resistive memory devices (e.g., memristors) to process synaptic operations in the analog domain. See sidebar.
The large scale of biological neural systems is already challenging the upper limits of CMOS electronics technologies, and thus the development of novel devices which can acheive the high densities and lower power operation will be critical for scaling neuromorphic systems to biologically-relevant scales. Resistive memory technology is uniquely suited for neural applications, as not only do they enable high density synaptic connectivty but also provide a mechanism for synaptic-like learning without substantial external control circuitry.
Severa W, Parekh O, Carlson KD, James CD, and Aimone JB – “Spiking Network Algorithms for Scientific Computing” – Proceedings of the IEEE International Conference on Rebooting Computing October 2016
Rothganger F – “Computing with dynamical systems” – Proceedings of the IEEE International Conference on Rebooting Computing October 2016
Vineyard CM and Verzi SJ – “Overcoming the Static Learning Bottleneck – the Need for Adaptive Neural Learning” – Proceedings of the IEEE International Conference on Rebooting Computing October 2016
Agarwal S, Plimpton SJ, Hughart DR, Hsia AH, Richter I, Cox JA, James CD, and Marinella MJ – “Resistive Memory Device Requirements for a Neural Algorithm Accelerator” Proceedings of the International Joint Conference on Neural Networks 2016
Draelos TJ, Miner NE, Cox JA, Lamb CC, James CD, and Aimone JB – “Neurogenic Deep Learning” Proceedings of the International Conference on Learning Representations 2016
Agarwal S, Quach T, Parekh OD, Hsia AH, Debenedictis EP, James CD, Marinella M, and Aimone JB – “Energy Scaling Advantages of Resistive Memory Crossbar Based Computation and its Application to Sparse Coding” Frontiers in Neuromorphic Engineering. January 2016
Cox JA, James CD, and Aimone JB – “A Signal Processing Approach for Cyber Data Classification with Deep Neural Networks” Complex Adaptive Systems - Procedia Computer Science. 61, November 2015
Vineyard CM, Verzi SJ, James CD, Aimone JB, and Heileman GL – “MapReduce SVM Game” INNS Conference on Big Data - Procedia Computer Science. 53, August 2015
Vineyard CM, Verzi SJ, James CD, Aimone JB, and Heileman GL – “Repeated Play of the SVM Game as a Means of Adaptive Classification” Proceedings of International Joint Conference on Neural Networks July 2015
Rothganger F, Evans BR, Aimone JB, and DeBenedictis EP – “Training neural hardware with noisy components” Proceedings of International Joint Conference on Neural Networks July 2015
Marinella MJ, Mickel PR, Lohn AJ, Hughart DR, Bondi R, Mamaluy D, Mjalmarson H, Stevens JE, Decker S, Apodaca R, Evans B, Aimone JB, Rothganger F, James CD, and Debendictis EP – “Development, Characterization, and Modeling of a TaOx ReRAM for a Neuromorphic Accelerator”, ESC Transactions, 64 (14) 2014
Lohn AJ, Mickel PR, Aimone JB, Debenedictis EP, and Marinella MJ – “Memristors as synapses in artificial neural networks: Biomimicry beyond weight change” In Cybersecurity Systems for Human Cognition Augmentation. 2014