Notes 20110127 CIS 6050 Neural Networks

Jump to: navigation, search

Back Propagation (BP) Advantages

momentum -- the first and arguably best training innovation
the most common method to accelerate convergence
from signal processing
add a momentum term to the weight change during training (ΔW_ij)
Δ_ij = αb_id_j
becomes ...
ΔW_ij(n+1) = αb_id_j + γΔW_ij(n)
where γ ∈ [0,1]
where n is the index for presentation (time -- exemplars -- data points)
γ is a constant which affects momentum rate
α -- learning rate → low [0.05, 0.1]
γ -- momentum rates → [0.8, 0.9]
engineering: is a filter
filter out high frequency variations in the error surface
this is a high pass filter -- where small variations are removed and big variations are retained
useful for weight spaces with long ravines with sharp curvature

-|      \
 -|      \
  -|      \
   -|      \
    -|      \
     -|      \

two-bit parity (XOR)
input → output
(0,0) → 0
(0,1) → 1
(1,0) → 1
(1,1) → 0
requires two inputs
one output
input -- two nodes
hidden -- two nodes
output -- one node
with learning rate of α=0.25, it took 6587 presentations of the data to learn (~1500 epochs)
where one epoch is a single complete presentation of the entire training set

problem: back propagation does not process sequential information
-- it keeps no history (context) of past events
a context needed for text conversion
- a in cake vs a in cat.
- i, th in with.
problem solved by presenting the surrounding six letters to the network at one time
creates a context in the inputs
sliding window in time
input[0] = char[-3]
input[1] = char[-2]
input[2] = char[-1]
input[3] = char[0]
input[4] = char[1]
input[5] = char[2]
input[6] = char[3]
input layer (6 units) → hidden layer → output layer
called windowing
each of the seven character input windows is a 29 component binary vector
input = ⟨ 26-letters, space, two punctuation chars ⟩
only one of the 29 inputs per window is set to one {1.0}
indicates which letter is active in that window
7 windows * 29 characters = 203 binary inputs
there were 26 outputs
output = ⟨ 23-phonemes, three features for stress and syllable boundaries ⟩

"the cat ran"
 the cat
  he cat 
   e cat r
     cat ra
     cat ran