I''ve got a question about recurrent Neural Networks. How do I fire signals through it? Such NNs can have several outputs, so should I divide signal into 3,5 for example and only then multiply by the weights? How should recursion be solved? Any good URLs or a brief explanation?

Hi,

I don''t really know what you mean but ordinary MLFF-NNs can also have multiple outputs.

/Mankind gave birth to God.

And what should I do if one output of a neuron results into a circle to itself(2->2, or indirect 2->3, 3->2, for example) and another goes to output. Accumulate infinite output value, if weights increase signal or perform one cycle and accumulate values "inside" neurons or what?

It doesn''t sound, frankly, like you have a grasp on standard feedforward neural nets yet - you should look into that. To answer your questions:

When you say "Such NNs can have several outputs" I am assuming you really mean that _neurons_ in such ANNs have several outputs. (This assumption is simply since your question makes no sence the other way -> almost all ANNs have multiple outputs...) This is quite simple to answer -> each ''output'' passes on the the value from the node unaltered.

Recursion should be solved by simply advancing each signal one timestep when a new input is given. (Note: it is possible to have all inputs ''off'' and continue moving the signals)

When a neuron sends a signal to itself (directly or indirectly) it will not accumulate an infinite value -> the value is passed though an activation function that will limit its responce to the bounds of the actvation function (normally 0 to 1 or -1 to 1).

Neurons do not store values. They simply sum values of inputs, perform the activation function on said sum, and pass the value on to all ''outputs.'' In implementation you may find it convienent to allow the neurons to store values but those values will be erased after one time step (or are, otherwise, useless).

In the end, your best bet will be a good book or a good website. I am not a search engine, and will therefore, not be able to recommend any websites. (Though if you search for recurrent neural networks you should find plenty of information).

- mongrelprogrammer

hi,

I think you just propagate the signals one step at a time.
Thus, at each time t the neurons propagate their signal so that these signals are available at time t+1.
That is, the net behaves according to this:

Xi(t+1) = G(W, I, X(t))

where Xi(t) is neuron i''s output at time t. W is weightvector, I is external inputvector. So each neuron is a function (G) of these.

Does this answer your question?

regards

/Mankind gave birth to God.

Yep, I''ve messed with "several outputs" and infinity, it''s just pretty late here and I was in a hurry=)

As for time-steps,- this is what I was asking for. Thanks for this tip. But if I advance everything one timestep every input-session, does that mean that if I have the shortest path to the output as 5 neurons in a row, I''ll get my first output after i feed 5 packs of input values? Or should one time-step last until I get the next output values ready? (Emm.. did I express myself correctly?)

Ehm, as already said: any good introductory text on RNN will probably have all the answers.

I think that you have to make, in your case, 5 timesteps/calcs/propagations to get the output for one individual inputpattern. Coz otherwise the last output neurons won't respond to the latest input.

(I think).

PS. hmm, that means you have to know the shortest path to the outputs. I haven't thought about that. I'll read up on RNNs when I have the time though...

/Mankind gave birth to God.

[edited by - silvren on January 1, 2003 4:11:38 PM]

quote: Original post by Halo Vortex
...if I advance everything one timestep every input-session, does that mean that if I have the shortest path to the output as 5 neurons in a row, I''ll get my first output after i feed 5 packs of input values? Or should one time-step last until I get the next output values ready?

It will take the net (in this case) 5 sets of data before any true results are given from the data. Most recurrant nets only have a disparity of one (the outputs are fed into the hidden units directly after the input neurons / only outputs are recurrant) and thus give results immediatly. I suppose if you really had to know how long this distance was you could use your favorite shortest-path algorithm (since ANNs are reasonably small for a shortest-path algorithm any of them would be a reasonable solution) and treat the ANN as a di-graph.

While I don''t have any of my references in front of me I can''t rember any ANNs that actually would require such a complicated recurrant setup. Any chance you could illuminate me as to what you are attempting? You have managed to make me curious...

- mongrelprogrammer

I''ve already implemented Multilayer FeedForwards, made Black&White like mouse-movement recognition working with BackProp, simple Genetic Evolution, so I thought about digging into Reccurent Networks. As everywhere they were descibed as difficult to train with math. algorithms, Genetic Evolution of structure and weights seemed to be a nice approach. I got one cool doc on their approach and in one of their test-results was shown a network, that had "back-connections" not only from the outputs, but from one hidden neurons to others, and such combination worked better with less hidden neurons that would be required otherwise.

Evolving the weights and the structure isn''t a big problem, although some of their advanced Evolution methods require quite a lot of coding. But until now I didn''t know how to launch data through such evolved nets.

The only question remaining is which of 2 ways to do time stepping?
1) 1 and 2 transfer their values to 3, 3 THEN transfers a new summed value to 4
2) 1 and 2 transfer their values to 3 and AT THE SAME TIME 3 transfers it''s unchanged value to 4

As for the shortest path, I don''t need to know it. By saying that I meant "number of timesteps until I get the first output". So don''t bother with it.
But it''s easy if anyone needs to know it: just launch a network and count the cycles until you get a real value in a target output. Number of cycles will be the shortest path.