Neural Net Problems - Exercise 4

Here are my solutions to exercise 4.

Implementing Our Network to Classify Digits

Part 1

Question

Write out $a^{'} = σ (w a + b)$ in component form, and verify that it gives the same result as the rule, $\frac{1}{1 + e x p ( - \sum _{j} w _{j} x _{j} - b )}$ , for computing the output of a sigmoid neuron.

Solution

Let it be stated that I have not yet taken linear algebra at college, so I have very limited experience with it.

Let us say that layer 2 has $2$ nodes and layer 1 has $3$ nodes.

The weights from layer 1 to layer 2 can be expressed as the following ( $w_{ji}$ , where $j$ is the neuron in the second layer and $i$ is the neuron in the first layer):

w = [w_{11} w_{21} w_{12} w_{22} w_{13} w_{23}] a = ⎣ ⎡ a_{1} a_{2} a_{3} ⎦ ⎤ b = [b_{1} b_{2}]

w a = [w_{11} a_{1} + w_{12} a_{2} + w_{13} a_{3} w_{21} a_{1} + w_{22} a_{2} + w_{23} a_{3}] w a + b = [(w_{11} a_{1} + w_{12} a_{2} + w_{13} a_{3}) + b_{1} (w_{21} a_{1} + w_{22} a_{2} + w_{23} a_{3}) + b_{2}] a^{'} = σ (w a + b) = [σ ((w_{11} a_{1} + w_{12} a_{2} + w_{13} a_{3}) + b_{1}) σ ((w_{21} a_{1} + w_{22} a_{2} + w_{23} a_{3}) + b_{2})]

This is exactly the same as $\frac{1}{1 + e x p ( - \sum _{j} w _{j} x _{j} - b )}$ but computed for both neurons at once with matrices!

Say we wanted to compute the output of the first sigmoid neuron in layer 2.

a_{1}^{'} = σ (j \sum w_{j} a_{j} + b) = σ ((w_{1} a_{1} + w_{2} a_{2} + w_{3} a_{3}) + b)

Also, I wanted to note that I found this website called the ml cheatsheet and it has been really useful in describing the mathematic concepts.

The header image was taken from Khan Academy.