If anyone has ever read a book called Prey by Michael Crichton, Is there any specific ai algorithm for "distributed processing" like the swarms in the book? Has anyone ever done it? The swarms should be able to evolve just like in the book. If you haven't read the book, then basically the swarms are nanoparticles which are programmed to simulate predators. Only fundamental rules are programmed into each particle and as a whole, the outcome is unpredictable.

Look up Particle Swarms.

A couple of hits from Googling "Particle Swarm"
http://www.swarmintelligence.org/
http://en.wikipedia.org/wiki/Particle_swarm_optimization

Emergent behavior all over again. Its interesting, (in the "watching ants" kind of interesting), and work well for simple cases, see flocking for an example. Such AI is of course completely reactive, and thus works only when the behaviors you desired are completely reactive too.

PS: I didnt read the book, but I dont think what the OP describe has anything to do with Particle Swarm Optimization, which is not much more than a multiple start-points gradient descent.