If you just want to experiment with neural networks and get the feel then use MATLAB, it has an excellent neural network toolbox with excellent documentation. And then afterwards you can translate it to your programming language of choice.

Quote: Original post by Predictor
The examples you give seem to require ignoring common practices in empirical modeling, such as feature selection.

I think they don't -- they just require that there are many features selected, rather than just a few. The plot I linked above -- the one with 1000 features -- that was after selecting the 1000 features from a much larger set of candidates.

Quote: Original post by Predictor
I'm not clear on why one would do so.

As I said, the main reason would be that in a given situation you may not know how to select a better set of features than the trivial set you already have.

Quote: Original post by Predictor
Further, I don't see how any of this indicates over-fitting.

Well, if you feed the network individual image pixels, and it works well on the training set but not on the test set, that typically indicates overfitting.

Quote: Original post by Gil Grissom

Quote: Original post by Predictor
The examples you give seem to require ignoring common practices in empirical modeling, such as feature selection.

I think they don't -- they just require that there are many features selected, rather than just a few. The plot I linked above -- the one with 1000 features -- that was after selecting the 1000 features from a much larger set of candidates.

Why select so many features when there are perfectly good feature selection procedures available?

Quote: Original post by Predictor
I'm not clear on why one would do so.

As I said, the main reason would be that in a given situation you may not know how to select a better set of features than the trivial set you already have.

One does not need to know which features are useful ahead of time any more than one needs to know regression coefficients ahead of time: That's what statistical procedures are for. There are any number of mechanical feature selection procedures which will solve this problem automatically.

Quote: Original post by Predictor
Further, I don't see how any of this indicates over-fitting.

Well, if you feed the network individual image pixels, and it works well on the training set but not on the test set, that typically indicates overfitting.

No, that is, by definition, poor fitting. Whether it is underfitting (most likely in the case you describe), overfitting or something else altogether is another matter.

I still have not heard any evidence to demonstrate that neural networks, in general, are "very difficult to train" or that they are "very prone to overfitting". Even were overfitting a problem, it is easily overcome through some form of regularization, such as early stopping.


-Will Dwinnell
Data Mining in MATLAB

Quote: Original post by Predictor
Why select so many features

In that particular case, this many features were necessary to achieve adequate performance.

Quote: Original post by Predictor
there are perfectly good feature selection procedures available?

There are any number of mechanical feature selection procedures which will solve this problem automatically.

I think you are overly optimistic. There are many cases in which all you have is the image (i.e., raw pixels), and there is no existing feature selection procedure which can take these pixels in and produce a sparse but useful set of features as an output.

But if you do know of such a procedure, I'd be thrilled to hear about it.

Quote: Original post by Predictor
One does not need to know which features are useful ahead of time any more than one needs to know regression coefficients ahead of time: That's what statistical procedures are for.

I think one does. Otherwise, one would end up with a huge number of potentially useful features. In the example above, I started with roughly 10 million candidates. Feeding them all to a neural network would be infeasible (especially given your comment that even 1000 features is "many").

Quote: Original post by Predictor
No, that is, by definition, poor fitting. Whether it is underfitting (most likely in the case you describe), overfitting or something else altogether is another matter.

Underfitting would imply poor training set performance, which wasn't the case in that example.

Quote: Original post by Predictor
I still have not heard any evidence to demonstrate that neural networks, in general, are "very difficult to train" or that they are "very prone to overfitting".

This is mostly my experience from dealing with a few large object recognition problems, plus the fact that I know of only one successful application of NNs in that area. But again, if you do have examples to the contrary, I think it would be interesting to discuss them.