I've been working solo on a robot combat game inspired by Robot Wars, Battlebots and such. I attached a clip from an older build of the game since showing is better than telling for describing stuff 
Players can choose a chassis, drivetrain and weapon for their robots before duking it out in the arena. This is where my problem comes in. I'd like to have a decently wide selection of parts with different appearances and different stats. I intend to allow players to use any combination of parts they choose. Since I'm working solo that's way, way more playtesting than I can reasonably achieve on my own or with the group of friends I could scrape together for such a thing.
I've already implemented a basic AI for the game so I'm considering using it to autonomously test out every possible pairing of designs against each other so I can see if any are especially good or bad. The game would collect stats like who won, damage taken, damage done etc. for each match and dump it all into a giant CSV for processing.
What I'm wondering is if anyone else has done something like this? And what sort of pitfalls I should be aware of before getting too deeply into AI vs AI playtesting? If anyone has any postmortems on this sort of automation I'd love to see them.
