Momentum
• When updating a weight, also add the
previous update to that weight times a
momentum constant m (0.0 <= m < 1.0).
• Possible to carry weights
– across plateaux in error surface
– through local minima to global minima
– through global minima to local minima (i.e. can
have undesirable effects as well).