quote:

---

Original post by Sneftel
I think my AI professor put it best: "ANNs are the second best way to do anything". It''s a useful techique, and the generality with which it may be applied is very attractive. But it''s a technique that cannot exploit the invariants and heuristics of a particular problem, and thus is usually doomed to be less effective than a custom-engineered AI solution.

---

Can you give an example of an invariant or heuristic that neural networks cannot exploit, which some other technique could?

-Predictor
http://will.dwinnell.com

quote:

---

Original post by Predictor
Do you mean that neural networks are never the best tools for such tasks, that neural networks are usually not the best or that neural networks are not always the best, or something else altogether?

---

Lock in ''b'' please! What I meant above is that ANNs are usually not the best task for the job. Why do I think that? Because, generally speaking, the best tool is job dependent... ANNs are a broad spectrum solution, usually tunable to a given task. That broad spectrum applicability comes at the cost of accuracy. Even in tasks where the ANN is more accurate than some other method, this accuracy comes at a higher price due to the requirement of data pre-processing, extensive tuning of network architecture and extensive training. I admit to a slight bias, but I believe that techniques such as MML classification are, while maintaining broad spectrum applicability, also typically more accurate than ANNs on the same problems. Personally, I''m a big fan of causal modelling.

Of course, I''m by no means an expert on ANNs. My beliefs arise from limited experience with ANNs (although a lot of experience with other techniques) and reading lots of literature, so if you can offer a counter claim based on your experiences, I''m all ears!

Cheers,

Timkin

<SPAN CLASS=smallfont>quote:

---

Original post by Timkin

<SPAN CLASS=smallfont>quote:

---

Original post by Predictor
Do you mean that neural networks are never the best tools for such tasks, that neural networks are usually not the best or that neural networks are not always the best, or something else altogether?

---

</SPAN>

Lock in ''b'' please! What I meant above is that ANNs are usually not the best task for the job. Why do I think that? Because, generally speaking, the best tool is job dependent... ANNs are a broad spectrum solution, usually tunable to a given task. That broad spectrum applicability comes at the cost of accuracy. Even in tasks where the ANN is more accurate than some other method, this accuracy comes at a higher price due to the requirement of data pre-processing, extensive tuning of network architecture and extensive training. I admit to a slight bias, but I believe that techniques such as MML classification are, while maintaining broad spectrum applicability, also typically more accurate than ANNs on the same problems. Personally, I''m a big fan of causal modelling.

Of course, I''m by no means an expert on ANNs. My beliefs arise from limited experience with ANNs (although a lot of experience with other techniques) and reading lots of literature, so if you can offer a counter claim based on your experiences, I''m all ears!

---

</SPAN>

I''d say there is a domain-knowledge spectrum along which problems lie. Some problems are well-studied and understood and have known, closed-form solutions.

The less that is known about a problem, especially regarding the functional form of its solution, the more useful empirical modeling becomes (assuming the availability of historical data of sufficient quality and quantity, of course). This is particularly true of nonparametric modeling, such as neural networks.

Assuming that a problem is a good candidate for empirical modeling, I know of no reason that neural networks (and remember that term covers many different techniques) should be considered, a priori, any better or worse than any other modeling technique.

-Predictor
Data Mining in MATLAB


[Edited by - Predictor on February 13, 2009 4:15:28 PM]

quote:

---

Original post by Predictor
Assuming that a problem is a good candidate for empirical modeling, I know of no reason that neural networks should be considered, a priori, any better or worse than any other modeling technique.

---

That''s a fair comment and one I would expect from someone who has made a living from data mining! I think one of the big detractions for ANNs though is that techniques for unsupervised learning of network topology are still in their infancy, even though ANNs have been widely used for more than 20 years in commercial practice. This results in the need for trained, specialised and experienced users who can craft networks by hand and tune them to produce answers. Those answers are then necessarily correlated with the practices of the user, meaning that one must believe in the quality of the user before they can believe in the quality of the results.

Cheers,

Timkin

<SPAN CLASS=smallfont>quote:

---

Original post by Timkin

<SPAN CLASS=smallfont>quote:

---

Original post by Predictor
Assuming that a problem is a good candidate for empirical modeling, I know of no reason that neural networks should be considered, a priori, any better or worse than any other modeling technique.

---

</SPAN>

That''s a fair comment and one I would expect from someone who has made a living from data mining! I think one of the big detractions for ANNs though is that techniques for unsupervised learning of network topology are still in their infancy, even though ANNs have been widely used for more than 20 years in commercial practice. This results in the need for trained, specialised and experienced users who can craft networks by hand and tune them to produce answers. Those answers are then necessarily correlated with the practices of the user, meaning that one must believe in the quality of the user before they can believe in the quality of the results.

---

</SPAN>

Regarding MLPs in particular, my feeling is that too much is made about discovering the "optimal" architecture. My experience has been that MLPs with some variety of sigmoid transfer function, a single hidden layer, and no fanciness (no jump connections, recurrent connections, node pruning, etc.) can be adequately be trained using early stopping to prevent overfitting.

While I agree that this work is best performed by a qualified analyst, I think that appropriate tools will make short work of neural network training. Contrary to the stories of neural network newcomers which I read on Usenet and forums such as this, my neural networks rarely require hundreds of hidden neurons or days to train. I suppose this leaves us in at least partial agreement that neural networks (MLPs, anyway) can be demanding to implement.

I do disagree with your last point, however. No one ever need take my models on faith, or guess as to how well they will work in practice. I test my models soundly using error resampling and (ignoring model drift) can tell you how well they will work using the performance metric of your choice. Good practice demands this sort of model assessment. Whatever idiosyncracies exist in my work, good or bad, will show up in the validation.

-Predictor
Data Mining in MATLAB

[Edited by - Predictor on February 13, 2009 4:54:09 PM]

quote:

---

Original post by Predictor
I do disagree with your last point, however. No one ever need take my models on faith, or guess as to how well they will work in practice. I test my models soundly using error resampling and (ignoring model drift) can tell you how well they will work using the performance metric of your choice.

---

That''s not quite what I meant. Certainly, any given model, or set of models, can be assessed for quality using an objective measure. However, the choice of the original model is still subject to the users experiences, or their ability to tune a given model. See my point? Certainly, an experienced user is going to hit the nail on the head more often than not and select a reasonable model... but a lay person is not. A lay person knowing that their model is inaccurate according to some measure doesn''t typically help them to discern a better model.

Cheers,

Timkin

<SPAN CLASS=smallfont>quote:

---

Original post by Timkin
Certainly, any given model, or set of models, can be assessed for quality using an objective measure. However, the choice of the original model is still subject to the users experiences, or their ability to tune a given model. See my point? Certainly, an experienced user is going to hit the nail on the head more often than not and select a reasonable model... but a lay person is not. A lay person knowing that their model is inaccurate according to some measure doesn''t typically help them to discern a better model.

---

</SPAN>

Mmm... I suppose MLPs can be more subtle than competing modeling technologies (tree induction, rule induction, k-NN, etc.), but I don''t know how much difference this makes in terms of needed skill, given the skill necessary to deal with all the issues outside of actual modeling (sampling, data pre-processing, data post-procesing, etc.) which will be needed for any modeling project. I have seen people wreck their comparatively simple linear and logistic regression analyses on the rocks of inadequate sample size, questionable data collection procedures and poor target variable definition.

-Predictor
Data Mining in MATLAB

[Edited by - Predictor on February 13, 2009 4:20:10 PM]

Yes, I''d agree that other techniques are not ''straightforward'' either. All data-mining algorithms that I have encountered required some level of prior knowledge of the technique and its applicability to arrive at reasonable results, even those based on causal modelling (although very little information was required in this case and only because the software was research grade, not commercial grade). I would suspect that preference for one technique over another, or rating as to the optimality of one technique over others, will always be subjective based on ones experiences, unless someone does develop the perfectly idiot-proof data-mining tool!

It''s been an interesting discussion. Thanks!

Timkin