Discussion about math, puzzles, games and fun.   Useful symbols: ÷ × ½ √ ∞ ≠ ≤ ≥ ≈ ⇒ ± ∈ Δ θ ∴ ∑ ∫  π  -¹ ² ³ °

You are not logged in.

## #1 2016-07-14 15:18:47

Agnishom
Real Member
From: Riemann Sphere
Registered: 2011-01-29
Posts: 24,838
Website

How does one avoid the problem of local minima in gradient descent?

'And fun? If maths is fun, then getting a tooth extraction is fun. A viral infection is fun. Rabies shots are fun.'
'God exists because Mathematics is consistent, and the devil exists because we cannot prove it'
I'm not crazy, my mother had me tested.

Offline

## #2 2016-07-14 15:23:41

bobbym
bumpkin
From: Bumpkinland
Registered: 2009-04-12
Posts: 109,606

Like Newton's, steepest descent can converge to minor valleys.

In mathematics, you don't understand things. You just get used to them.
If it ain't broke, fix it until it is.
Always satisfy the Prime Directive of getting the right answer above all else.

Offline

## #3 2016-07-14 15:42:17

Agnishom
Real Member
From: Riemann Sphere
Registered: 2011-01-29
Posts: 24,838
Website

One solution is momentum. How do I mplement that?

'And fun? If maths is fun, then getting a tooth extraction is fun. A viral infection is fun. Rabies shots are fun.'
'God exists because Mathematics is consistent, and the devil exists because we cannot prove it'
I'm not crazy, my mother had me tested.

Offline

## #4 2016-07-14 15:57:17

bobbym
bumpkin
From: Bumpkinland
Registered: 2009-04-12
Posts: 109,606

A steepest descent?

In mathematics, you don't understand things. You just get used to them.
If it ain't broke, fix it until it is.
Always satisfy the Prime Directive of getting the right answer above all else.

Offline

## #5 2016-07-14 16:05:17

Agnishom
Real Member
From: Riemann Sphere
Registered: 2011-01-29
Posts: 24,838
Website

No, momentum

'And fun? If maths is fun, then getting a tooth extraction is fun. A viral infection is fun. Rabies shots are fun.'
'God exists because Mathematics is consistent, and the devil exists because we cannot prove it'
I'm not crazy, my mother had me tested.

Offline

## #6 2016-07-14 16:32:49

bobbym
bumpkin
From: Bumpkinland
Registered: 2009-04-12
Posts: 109,606

The only thing I know about momentum is that it is mass x velocity.

Take a look at this demo:

http://demonstrations.wolfram.com/Optim … aFunction/

In mathematics, you don't understand things. You just get used to them.
If it ain't broke, fix it until it is.
Always satisfy the Prime Directive of getting the right answer above all else.

Offline

## #7 2016-07-15 00:58:25

Agnishom
Real Member
From: Riemann Sphere
Registered: 2011-01-29
Posts: 24,838
Website

Momentum is often used with gradient descent to make sure that the solution does not get stuck in a local minima.

'And fun? If maths is fun, then getting a tooth extraction is fun. A viral infection is fun. Rabies shots are fun.'
'God exists because Mathematics is consistent, and the devil exists because we cannot prove it'
I'm not crazy, my mother had me tested.

Offline

## #8 2016-07-15 01:49:38

bobbym
bumpkin
From: Bumpkinland
Registered: 2009-04-12
Posts: 109,606