Hi, I'm a bit confused about the bias value in a feedforward neural network. I've just read through the comp.ai.neural-nets FAQ and also looked at several tutorials prior to that. What I understand is that the bias is connected to every node after the input layer and none in the previous layer. I've read that the bias value is always '1' and this is where the confusion arises... Does that mean the bias value, when multiplied by its connection weight, always emits a '1' or the value of bias before multiplication with its connection weight is '1'? bias * weight = 1 or bias = 1; => 1 * weight; Thanks for any help :) Max

The latter.

Thanks! :)

Saying that the bias value is 1 but it is multiplied by an arbitrary connection weight is a way to formally give the network an uniform structure; what it really means is that each node has its own bias, and it is treated like regular weights for the purposes of normalization, training, etc.

Quote:
...and it is treated like regular weights for the purposes of normalization, training, etc.

Thanks for your reply LorenzoGatti, so what you are saying is that the bias should be treated like the other weights in the network and be evolved if for example the network was being trained using a genetic algorithm, rather than just being a static random variable assigned to each neuron at the start of training?

I thought that the formula went like this:

Sigmoid(input * weight + bias) = output

Or whatever if you're not using a sigmoid function. I thought bias was treated as another weight just for training purposes, but it was added to the activation value, not multiplied with.

All of these formulations are equivalent. I personally prefer to think about biases as an extra input that has a fixed value of 1. In classical linear regression it is usually called "intercept".