I'm currently modeling a flame using 3DS Max's particle system. I want to use this flame for a 3D game that I'm developing. Recently I have realized that modeling a flame and exporting it to Unity (or any other game/graphics engine) is not as simple as it seems. My question is: After modeling the flame, what do I have to do to make it work properly in a game engine?

I'm asking this because my school teacher wasn't very clear on the matter when I asked him. He mentioned something about having to turn the flame object into a video clip of some sort and he showed me the menus where I had to do it, but I didn't understand most of what he said so I decided to come here with the hope of finding a better, and more productive, answer.

Usually particle systems are made as part of the game engine tool-set and not in an external editor.

According to your teacher's method the most likely work-flow -
Render the flame system as video clip or sequence of images in front view.
Create a bill-board in your game engine and apply this animated texture(s) / video clip.

Note: For the above method to work, the engine you are using should support video as particle materials. If it doesn't most probably it wont support GIF types either. So you need to have a timer for the billboard and keep cycling through the image set.