Bizar Artificial Neural Network

Started by Scrut July 12, 2010 03:41 AM

5 comments, last by Scrut 14 years, 7 months ago

Author

July 12, 2010 03:41 AM

I have a Neural Network that will not fire the correct outputs. The network output value will be approximately the same value regardless of the network input values. The network output will vary by a small amount I believe it was something around ~0.001. The network has input layer populated with 2 neurons, 2 hidden layers with 4 neurons, and the output layer contains 1 neuron. I have used two different implementations of the BackPropagation Algorithm used for the Training Algorithm; First BP Algorithm adjusted weight values, while the second BP Algorithm adjusted weight and bias values. Both with the same issue. I am attemping to train the bitwise exclusive or, the XOR operator.

you can download my source code here
Click on here for Source Code

Here is my training set i ran thru each sample in the set 10 times, so the network generation is 40
|--First Input Neuron---|---Second Input Neuron----|---Correct Output---|
|--------0.0-----------|----------0.0--------------|-------0.0----------|
|--------1.0-----------|----------0.0--------------|-------1.0----------|
|--------0.0-----------|----------1.0--------------|-------1.0----------|
|--------1.0-----------|----------1.0--------------|-------0.0----------|

And here is my code I am using implementing the backpropagation training algorithm and I knew 0 calculus so i bet its prob the training algorithm.

void BackPropagation( CNeuralNetwork *pcNeuralNetwork, double *pdbExample, double dbMomentum, double dbLearningRate ) {  int iOutputLayerID = pcNeuralNetwork->ReturnNeuralNetworkLayerCounter() - 1;  int i, k, j        = pcNeuralNetwork->ReturnNeuralNetworkLayer( iOutputLayerID ).ReturnNeuronCounter();	  for ( i = 0; i < pcNeuralNetwork->ReturnNeuralNetworkLayer( iOutputLayerID ).ReturnNeuronCounter(); i++ ) {    pcNeuralNetwork->ReturnNeuron( iOutputLayerID, i ).SetDelta( pcNeuralNetwork->ReturnNeuron( iOutputLayerID, i ).ReturnOutput() * ( 1.0 - pcNeuralNetwork->ReturnNeuron( j, i ).ReturnOutput() ) * ( 1 - pcNeuralNetwork->ReturnNeuron( iOutputLayerID, i ).ReturnOutput() ) * ( pcNeuralNetwork->ReturnNeuron( iOutputLayerID, i ).ReturnOutput() - pcNeuralNetwork->ReturnNeuron( iOutputLayerID, i ).ReturnOutput() ) );    for ( j = iOutputLayerID - 1; j > 2; i-- ) {      for ( k = 1; k <= iOutputLayerID; k++ ) {        pcNeuralNetwork->ReturnNeuron( j, k ).SetDelta( pcNeuralNetwork->ReturnNeuron( j, k ).ReturnOutput() * ( 1 - pcNeuralNetwork->ReturnNeuron( j, k ).ReturnOutput() ) * pcNeuralNetwork->ReturnNeuron( j + 1, i ).ReturnWeight( k ) * pcNeuralNetwork->ReturnNeuron( j + 1, i ).ReturnDelta() );      }    }  }  for ( i = iOutputLayerID; i > 2; i-- ) {    for ( j = 1; j < pcNeuralNetwork->ReturnNeuralNetworkLayer( i ).ReturnNeuronCounter(); j++ ) {    pcNeuralNetwork->ReturnNeuron( i, j ).SetBias( pcNeuralNetwork->ReturnNeuron( i, j ).ReturnBias() + ( dbLearningRate * 1.0 * pcNeuralNetwork->ReturnNeuron( i, j ).ReturnDelta() ) );    for ( k = 1; k < pcNeuralNetwork->ReturnNeuralNetworkLayer( i - 1 ).ReturnNeuronCounter(); k++ ) {      pcNeuralNetwork->ReturnNeuron( i, j ).SetWeight( k, pcNeuralNetwork->ReturnNeuron( i, j ).ReturnWeight( k ) + ( dbLearningRate * pcNeuralNetwork->ReturnNeuron( i - 1, k ).ReturnOutput() * pcNeuralNetwork->ReturnNeuron( i, j ).ReturnDelta() ) );    }   }  }}

If will respond with any additional info. Thanks for your help.

Cygon

1,219

July 12, 2010 06:44 AM

That is one huge unreadable line there :>

I can't spot anything obvious but I've written set of neural net classes 6 years ago, using the same concept (neurons in layers und training through backpropagation). If you wish, you can compare the code to see if you can find any differences:

neuralnet.cpp, neuronlayer.cpp, neuron.cpp

The relevant methods are NeuralNet::train(), NeuronLayer::backpropagateError() and Neuron::computeError().

Good luck!

Professional C++ and .NET developer trying to break into indie game development.
Follow my progress: http://blog.nuclex-games.com/ or Twitter - Topics: Ogre3D, Blender, game architecture tips & code snippets.

alvaro

21,610

July 12, 2010 06:51 AM

You should replace your identifiers with shorter, more readable ones. `ReturnNeuronCounter()' could simply be `num_neurons()' or even `size()' and everyone would understand what it's doing.

You should also store parts of your expressions in local variables, especially those that appear several times and can be given reasonable names.

When I started doing those two things in your code, some problems became obvious. For instance, this line:

    pcNeuralNetwork->ReturnNeuron( iOutputLayerID, i ).SetDelta( pcNeuralNetwork->ReturnNeuron( iOutputLayerID, i ).ReturnOutput() * ( 1.0 - pcNeuralNetwork->ReturnNeuron( j, i ).ReturnOutput() ) * ( 1 - pcNeuralNetwork->ReturnNeuron( iOutputLayerID, i ).ReturnOutput() ) * ( pcNeuralNetwork->ReturnNeuron( iOutputLayerID, i ).ReturnOutput() - pcNeuralNetwork->ReturnNeuron( iOutputLayerID, i ).ReturnOutput() ) );

became

    Neuron & output_neuron = nn->neuron(output_layer_id, i);    double output = output_neuron.output();    output_neuron.SetDelta(output * (1.0 - nn->neuron(j, i).output()) * (1 - output) * (output - output));

Since you are multiplying by `(output - output)', this is a convoluted way of setting delta to 0. Probably not what you wanted.

You are also initializing `j' to a value that never gets used, you never use two of your parameters, the loop that initializes and checks `j' decrements `i'...

There are too many things wrong with the code. You can't just write a whole bunch of code and then complain about the highest level behavior not working, when you haven't verified that any of the intermediate things are being done correctly. You have to test each little thing individually, and not write any more code until you got that right.

Also, learn how to use a debugger. It would have told you that delta is being set to 0, for instance, and it may have made you realize that the looping is messed up.

Emergent

982

July 12, 2010 09:05 AM

Since the graphs that represent ANNs are typically so dense, I wouldn't even bother with all this OO stuff for representing them; the only object you really need to keep around and manipulate is a weighted adjacency matrix. I.e.,

const int nNeurons = 10;double w[nNeurons][nNeurons];//...

or the equivalent with nicer array classes.

In other words, my personal opinion is that there's no point extending the (already strained) biological metaphor to your code when all you really need is a matrix (which I would probably make a private member of an ANN class, which would contain evaluation and training methods).

Plus, think about what you're doing in your current representation: You've got pointers and indirection everywhere, taking up 4 bytes of memory at a pop and scattered all over the place through a billion calls to new. The matrix representation, by comparison, leaves all connectivity information implicit, and takes up just one nice contiguous block of memory. Seems better all-around for a dense graph.

Scrut

Author

July 13, 2010 12:58 AM

yeah i am making two more, one with simpler OOP and with as few function calls as possable thus less jumps, and clump it up together then optimize it prob do forward propagation in asm should be easy. anyways i know the oop design is retarded to say the least to much really

Scrut

Author

July 13, 2010 09:36 PM

FINALLY first neural network trained! Cannot begin to explain the amount of errors. Thanks for your responses everyone. Cygon thanks for the code posted, because your code explained really well what was happening in the backpropagation algorithm. I really cant believe I got it working with as litle as I know about neural networks. Later.

[Edited by - Scrut on July 14, 2010 12:36:45 AM]

Scrut

Author

July 15, 2010 07:05 PM

Here is the code for the Neural Network which finally completed training. It preforms multiplication; not as exact as I would like still experimenting with learning rate and momentum which unexpectedly, but obvious really, has dramatic effects on the over all.

Simple Artificial Neural Network

Bizar Artificial Neural Network

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Bizar Artificial Neural Network

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines