Notes 20110217 CIS 6050 Neural Networks

From SnOwy - Ed's Wiki Notebook

An application of self-organizing maps

• horse gaits -- when a horse is starting to go lame (is getting sick)
• measuring the strain on each of a horse's hooves at 30~60Hz
• five measurements per hoof: strain in two directions with magnitude
• didn't end up getting the characteristics for lameness BUT we got very nice results clustering horse gaits
• principle component appears to be the horse
• next important components: speed of gait, shod (shoes) or not

Hopfield Networks

• single layer of nodes
• autoassociative (content addressable memory CAM)
• given a partial pattern, tries to reconstruct original pattern (noise)
• each node is connected to all others but not itself
• able to represent a matrix with zeros on the diagonal
• able to build one with analogue continuous timesteps
• active node value is +1
• inactive node value is -1
• training ...
• weights are set once and not changed
• given M pattern, α₁, α₂, ... α_m of N dimensions (length)
• weights are calculated using ...
• w_ji = N^-1Σ_μ=1^M α_μj α_μi, j≠i
• w_ji = 0, j=i
• i,j are node indices
• α_μj is pattern μ and element j is in the pattern
• recall ...
• present a partial pattern and retrieve a full pattern ...
• requires iteration - calculate node values (activation) in random order until the system stabilizes
• pattern which is left in the nodes after it stabilizes, should be the training pattern without noise
• activation function ...
• x_j(t+1) = sgn( Σ_i=1^N w_jix_i(n) ), j=1,2, .. N
• where sgn(a) = a <=> 0
• -- activation in node j is based on the activation of all other nodes multiplied by the weights attached to node j
• calculated for all N nodes
• apply the formula until x_j does not change for j=1 ... N
• does not do classification
• does not do clustering
• very limited storage capacity
• number of patterns that can be stored almost without errors is ...
• M_max = N (2lnN)^-1
• if N = 10, M_max -- store 2.2
• if N = 20, M_max -- store 3.3
• if N = 200, M_max -- store 18.9
• if N = 400, M_max -- store 33.4
• if M_max is exceeded, then patterns are not recoverable
• solutions -- have "hidden" nodes, that is -- nodes that are trained with just zeros and whose outputs are not read
  • not exactly -- the window of unwritten/unread nodes actually moves around
  • what if we added redundant connections?
• number of patterns is divided by a logarithm of the length -- grows very slowly
• number of patterns linked to the length of the pattern so we must store longer patterns if we have a lot of them