At hobbyking.com you can find lots of cheap drone parts, including several flight controllers which are open source.
They are usually based on the Arduino platform.
There are several open source projects that support the flight controllers, multiwii was the first one (works with any arduino, and a wii nunchuck).
Several arduino-based flight controllers are now sold that contain a CPU and all the sensors you need.
Baseflight is based on multiwii, supports (at least one) flightcontroller with 32 bit arm processor (normal arduino is 8 bit AVR)
Cleanflight is another project, based on baseflight with support for more hardware.
I don't know of one where your phone is the controller though, but your probably could make that work through some telemetry solution (I know of some you can at least easily connect to Android phones)
In any case, you don't want your phone to do the actual flight controller calculations, that would be way too much lag, you need that in realtime, on the drone, so you will need a flight controller on the drone. (also, strapping the phone to the drone would be too heavy, unless you build a big drone, which is expensive)
You can then control the flight controller through a serial connection. The telemetry I mentioned is serial-over-radio.
Wlan could also work, there are small cheap wlan-chips you can buy from places like Adafruit and Sparkfun.
A problem with wlan though is the latency, but should be ok for a slow drone
I doubt you will find anything that works right out of the box though, it will take some programming of your own and lots of research.
100$ should be possible, for a small one, but it will not be easy to do. (as in, just buying a complete thing with everything and an SDK), you will need to buy it in parts and construct it yourself.
... or just wait, I'm sure someone will release something like you want within a year or two :P