So I hear alot about neural nets, but I almost never here about any other kinds of function approximators. Why is that? Are neural nets just better than everything else, or are the other methods so complex that nobody wants to use them? I've been working on a few different types of function approximators, but every time I try to find information about methods similar to mine I can't find anything. Most of my methods are so simple that I highly doubt they're original. For instance in one of my programs I use a weight matrix, but rather than just using the input I give it a constant, the input, each of the inputs squared and cubed etc. This allows it to approximate with polynomials. If I use several of these systems that each apply only in a given input range then I can approximate with piecewise polynomial equations. In another program I store a training set and then interpolate between that set of points to form the function. To me this seems like one of the most obvious things you could do for supervised learning, but I can't find any information on methods that do this. However, I have no idea what my methods would be called so that probably contributes to why I can't find any information.

Quote: Original post by Alrecenk
So I hear alot about neural nets, but I almost never here about any other kinds of function approximators. Why is that?

Because artificial neural networks became popular in the early 80s as the supposed panacea for AI and machine learning. They had some benefits over statistical learning methods, but ultimately failed to live up to the hype (and are now just one of the tools you can throw at a problem). However, they're widespread coverage in the media for several years meant there was a subsequent rush to publish books on them. A multitude of 'experts' got their opinion into print... and later online, meaning that when people go looking for information on function approximation, they usually come across a description of an ANN. Since they're trivially simple to code many people go no further, much to the amusement of those who know some functional calculus.

Quote:
I've been working on a few different types of function approximators, but every time I try to find information about methods similar to mine I can't find anything. Most of my methods are so simple that I highly doubt they're original.

If you want a detailed account of function approximation, start by reading up on Functional Calculus and the Spectral Theory of functions. If you can get a handle on these areas, then you'll understand the inner workings of almost all of the subsequent methods for function approximation.

Quote:
In another program I store a training set and then interpolate between that set of points to form the function. To me this seems like one of the most obvious things you could do for supervised learning, but I can't find any information on methods that do this.

It's out there... what you're describing is a very simple basis function network (you're bases are constants taken from the training surface). It's interpolation properties would not be particularly good on nonlinear functions, but it'd be fine for linear or quasi-linear surfaces. A very simple yet powerful extension of this is the Radial Basis Function network.

Cheers,

Timkin

You might want to look into Genetic Programming. One of the uses of it was for function approximation. I read a few papers where they used GPs to do wave form matching (given a wave form, find a function that closest approximates it).

If NNs are the king of overhyped function approximators, GAs are the king of overhyped global optimization algorithms.

Absolutely Sneftel!!!

Personally, my position is that if you want to understand function approximation, you need to understand functions first. If you don't know functional analysis, how do you expect to know whether or not what you are doing is sensible, or embodies the properties that you require of your approximation!?

Anyone can pick up a hammer and hit things with it, or pick up a saw and cut things... but making a fine dining table requires a lot more knowledge than how to cut wood and stick it back together again.

Timkin

Quote: Original post by Sneftel
If NNs are the king of overhyped function approximators, GAs are the king of overhyped global optimization algorithms.

Yes, that is very true. And it also shows that it really takes about 30 - 40 years before an AI/EC method to really hit main stream to the point that everyone tries to use it for everything. Given that track record, I guess most swarm algorithms for function optimization will take another 15 - 20 years.

Just to clear things up, in case someone misunderstands, GP and GA are different in many ways, and I know a few people who get offended if you mix them up. GAs were designed to do function optimization and what not, but GPs were designed to evolve solution strategies, which can be used as a function approximator because it will spit out a complete fiinction, not just numbers or weights. In many cases, since GP was designed in LISP, they have been used to evolve entire programs. In many ways, GP is similar to GAs, but its goal is lot grander in scale.

Quote: Original post by Timkin
Anyone can pick up a hammer and hit things with it, or pick up a saw and cut things... but making a fine dining table requires a lot more knowledge than how to cut wood and stick it back together again.

But let's not forget that many do not need a fine dining table, only a flat surface they can place a plate upon. :) Neural nets and genetic algorithms are certainly overhyped, but there is definitely something to be said for simple, well-known approaches over more accurate solutions that are more complex to understand and implement.

Its been said that NN's (and other hill climbers such as GA's) are always the 2nd best method.

That idealy you usualy want the best (1st) solution, or the most general (2nd)

Quote: Original post by WeirdoFu
Just to clear things up, in case someone misunderstands, GP and GA are different in many ways, and I know a few people who get offended if you mix them up.

Now, that we disagree on. GP is simply a particular application of GAs, and IMO it's subject to the same sorts of problems that all GA applications face--and more, because there's no clear consensus on how to represent the genome. GP might be useful down the road, but as far as I can tell the current level of adhockery makes it much less reliably useful than other, more mature ML strategies.

EDIT: One point in GPs' favor that I should mention is that in the realm of real-valued outputs they fare much better against their competitors.

Quote: Original post by Sneftel

Quote: Original post by WeirdoFu
Just to clear things up, in case someone misunderstands, GP and GA are different in many ways, and I know a few people who get offended if you mix them up.

Now, that we disagree on. GP is simply a particular application of GAs, and IMO it's subject to the same sorts of problems that all GA applications face--and more, because there's no clear consensus on how to represent the genome. GP might be useful down the road, but as far as I can tell the current level of adhockery makes it much less reliably useful than other, more mature ML strategies.

EDIT: One point in GPs' favor that I should mention is that in the realm of real-valued outputs they fare much better against their competitors.

I know this debate can go on for a bit, but I will just say this one last thing about GPs. There is a consensus on problem representation for GPs and it is tree structures. So, for function approximation, a GP is actively constructing and evolving entire functions, variable placement, coefficients, constants, operators etc. Of course, you can probably create something similar using a GA, but it would become rather complex and you will still end up with a GP as you veer away from the traditional GA genome, which is usually an array of some sort.

Fundamentally, you can say GAs and GPs are the same as they both use a population and apply crossover and mutation operators on the population, but their original design principles are very different. Where GAs were originally design as a general purpose problem solver, GPs were conceptualized as a means of evolving and creating problem solving strategies. One is more concerned with getting a good solution, while the other is more concerned about the process.