How to Speed Up Neural Network Training Using Momentum

A 1988 method for training artificial neural networks that uses an 'activating variable' to speed up how quickly the network learns from its mistakes.

What it covers

The patent describes a way to train neural networks by splitting each neural unit into two parts. The first part handles the primary learning, while the second part tracks changes over time to create an 'activating variable.' This variable acts like momentum; it is added to the feedback signal to accelerate how quickly the network adjusts its internal variables (weights) to reach a correct solution. By comparing the network's output to a desired target and iterating through examples, the system converges on a result faster than standard methods of the time.

What it doesn't cover

• —Does not cover modern deep learning architectures like Transformers or Convolutional Neural Networks.
• —Does not cover hardware-specific implementations like GPUs or TPUs.
• —Does not cover unsupervised learning techniques where no desired output is provided.
• —Does not cover backpropagation algorithms that do not utilize this specific two-subunit momentum mechanism.

The clever bit

The invention uses a dual-subunit structure where one subunit operates with a longer time constant than the other, effectively creating a 'memory' of past updates to accelerate future ones.

Why it matters

This patent represents an early attempt to solve the 'slow learning' problem in artificial neural networks during the late 1980s. It predates the modern deep learning boom and highlights the historical focus on optimization techniques that remain foundational to how we train AI today, specifically the concept of momentum in gradient descent.

Real-world examples

• 1.Early artificial neural network research software
• 2.Gradient descent optimization algorithms with momentum