Notes 20110113 CIS 6050 Neural Networks

From SnOwy - Ed's Wiki Notebook

Contents 1 Introduction to Neural Networks 2 Clustering 3 Classification 4 Components of ANN 5 Weights (connections, synapses) 6 Processing Elements (nodes, units, PE's, neurons) 7 A single node 8 The Inputs and Weights 9 The Processing Node 10 The Activation Function (threshold function) 11 Typical Activation Functions 11.1 Threshold Function (step, bipolar) 11.2 Piecewise Linear (non-linear ramp) 11.3 Sigmoid Function (e.g. logistic, e.g. tanh) 12 Node Organization 13 Setting the Weights 14 Interconnection Schemes

Introduction to Neural Networks

• overview -- what are they used for?
  • classification
  • clustering
  • modelling real neurons (not in this class -- connectionism)
• non-linear problems

Clustering

• input data x → ANN → a map
• a neural network builds a map so that input data is clustered -- a topologically consistent formation
• things that are similar are closer together; we can figure when these things are different

Classification

• associates a label or a name with an input
• input data x → ANN → output label
• a very simple example: inputs are temperature in °C → ANN → state of water
• classifying sequences
• an interesting example: < 45, 46, 50, 53 > → ANN → increasing

Components of ANN

• architecture: weights and processing units
• rule to ...
  • set the weight values
  • store and retrieve information

Weights (connections, synapses)

• connect two processing units
• change values during training
• often a two-dimensional array

Processing Elements (nodes, units, PE's, neurons)

• receive inputs from the weights
• often a one-dimensional array

A single node

• and associated weights
• Inputs < I₁, I₂, I₃ > &rarr < W_k1, W_k2, W_k3 > → node (with bias w) → v_k → f(v_k) → o_k (output)

1. the inputs I_i and weights
2. the node with bias w
3. the net activation, transfer function and output o_k

The Inputs and Weights

• I₁w_k1, I₂w_k2 ... = IW
• where I is a 1D array, and W is a 2D array
• in some cases, the calculation is a branching operation: choose the maximum of the input and weight

The Processing Node

• result of IW is summed, creates V_k
• v_k = Σ_iI_iw_ik
• bias value is also added
  • the bias is normally a threshold value

The Activation Function (threshold function)

• scales the net activation to something useable by limiting the range
• without this, the output magnitude could be large
• normally bounded by [0, 1] or [-1, 1]
• generally non-linear
• O_k = f(Σ_iI_iw_ki)

Typical Activation Functions

Threshold Function (step, bipolar)

• f(x < 0) = 0j
• f(x ≥ 0) = 1j
• where j in real

Piecewise Linear (non-linear ramp)

• f(x < -1i) = 0j
• f(x ≥ -1i or x ≤ li) = x
• f(x > 1i) = 1j
• where i, j in real
• a simplification of the sigmoid function

Sigmoid Function (e.g. logistic, e.g. tanh)

• S-shaped, bounded, monotonic, non-increasing
• monotonic stipulates that the function may not repeat the same range value
• non-increasing stipulates that the same value of the range may repeat
• logistic: (0, 1) -- S(x) = (1+e^-x)^-1
• tanh: (-1, 1) -- S(x) = tanh(x)
• not all networks use a non-linear activation function -- some use sums only

Node Organization

• organized in layers
• input layer -- receives signals from the environment
• output layer -- sends signals or results out to the environment
• hidden layer -- receives signals from a neural layer, sends signals to a neural layer; no direct connection with environment
• generally, all processing is completed in one layer before the results are passed to the next layer
• less often, nodes are considered as a bunch -- i.e. not in layers
• a wild tangle of nodes connected to one another
  • the Boltzmann machine

Setting the Weights

• the credit assignment problem
• a difficult problem

Interconnection Schemes

• the patterns for which the weights are connected
  1. intrafield (lateral) connections
     • between nodes of the same layer
     • often just called lateral
  2. interfield connections
     • between nodes of different layers
     • often just called connections (default)
     • can be one or two directional (can have different weights or two sets of weights)
     • completely connected: when all of nodes in one layer are connected to all nodes in the next layer (bipartite)
  3. recurrent connections
     • connects a node to itself