PerceptronPractice

Perceptron Practice

• Perceptron Notes
• Multi-Layer Perceptron Notes

Introduction

The Truth Table Problem (Logic AND, OR and XOR)

Data:

• Linear seprable: AND, OR => Single Layer Perceptron (SLP)
• Linear unseprable: XOR => Multi Layer Perceptron (MLP)

The Iris

Truth Table Problem

AND, OR Model

Structure:

Step Function

$$ Y = \left{ \begin{matrix} 1,\text{if }\sum_{i}^{}{a_{i}W_{i} \geq \theta} \ 0,\text{if }\sum_{i}^{}{a_{i}W_{i} < \theta} \ \end{matrix} \right. $$

Code:

• AND and OR

Result of AND

Training AND with one epoch of data (i.e. AND's truth table)

===== AND with init weight =====
Perceptron with
Current weight
 [ 0.2 -0.3]
Current bias
 0.4

After round 0 training
updated weight [ 0.2 -0.3] bias 0.4
After round 1 training
updated weight [ 0.2 -0.3] bias 0.4
After round 2 training
updated weight [ 0.2 -0.3] bias 0.4
After round 3 training
updated weight [ 0.3 -0.2] bias 0.30000000000000004


After training
Perceptron with
Current weight
 [ 0.3 -0.2]
Current bias
 0.30000000000000004

Result of OR

Training OR with one 100 epoch of data (i.e. OR's truth table) and Show its prediction result

===== OR with random weight =====
Perceptron with
Current weight
 [-0.16267954  0.34931872]
Current bias
 0.697259461774919

Perceptron with
Current weight
 [0.23732046 0.54931872]
Current bias
 0.19725946177491913

Predict
[0 0] --> 0.0
[0 1] --> 1.0
[1 0] --> 1.0
[1 1] --> 1.0

XOR Model 1

Structure:

Code:

• XOR

XOR Model 2

Another Structure

TBD

Iris

I have imitate the Keras API and reference from some of the code of eriklindernoren/ML-From-Scratch Github repository.

Model

Result of MLP

with verbose = False

• Measure the accuracy of the test subset (30% of instances)
• Training the model with 10 epochs and 32 batch size
• Using Cross Entropy as loss function

Accuracy of Keras with Adam optimizer is 0.8444444470935397
Accuracy of Keras with RMSprop optimizer is 0.5555555562178294
Accuracy of MLP From Scratch with Adam optimizer is 0.8064903846153846
Accuracy of MLP From Scratch with RMSprop optimizer is 0.984375

Model	Accuracy	Optimizer
(Multi-Layer Perceptron using Keras)	0.8444	Adam
(Multi-Layer Perceptron using Keras)	0.5555	RMSprop
Multi-Layer Perceptron From Scratch	0.8065	Adam
Multi-Layer Perceptron From Scratch	0.9844	RMSprop

Statistics Machine Learning on Iris From Scratch: Logistic Regression, SVM

Resources

Book

Deep Learning

• Ch 6 MLP
  • Ch 6.1 Example: Learning XOR

Article

• Solving XOR with a single Perceptron
• How To Implement The Perceptron Algorithm From Scratch In Python
• 類神經網路 筆記

Matlab

• Solving XOR problem with a multilayer perceptron

Slides

• Single Layer Perceptrons

Example

• eriklindernoren/ML-From-Scratch - Multilayer Perceptron
  • neural_network.py
  • layers.py
  • activation_functions.py
  • optimizers.py
  • loss_functions.py
• Keras - Getting started with the Keras Sequential model