Where do I purchase an old Atari plus empty cartridges? I'll also need information about writing games for it. Here are screenshots for an old-school game series that I'd love to re-program on an Atari. 

Which Atari? For a 2600 this game seems a lot too complex, graphics wise. The 2600 has very limited drawing capabilities and is really difficult to program for (you need to program 6502 assembly and get the timing exactly right, to the precision of a single clock cycle of the CPU to get a the result you want).

If you have to ask here instead of being able to find this information in Google, I'm afraid you may not be ready for this. Maybe start with an easier target? Like C64 or DOS?

You can use emulators for development, e.g. Stella for 2600. Probably much more comfortable.

The game shown looks like a vector graphics game, they required special hardware (I think Vectrex was the only capable console for home.)

To learn programming i'd not recommend old hardware but instead create retro game just on PC (you learn a lot faster, you can use high level language, you don't need to know every tiny bit about the system).

Alright. Is compiling for PS4 even something I should worry looking into? It's probably nothing more than a dream that Sony would give their allowance to program for one of their systems to a freelance game programmer like me, right?

Addressed much of this in your other, parallel thread. 

Sony is perfectly willing to license a devkit to you. 

First you must show that you are a competent, experienced developer, who has the ability to use the devkit without bothering people when things to don't work.  Then you need to demonstrate that you have an actual use for a devkit, meaning a high likelihood of actually completing a commercially viable game, including the process of going through product certification.  Next, you need to meet their requirements for having a kit, such as secure facilities and whatnot. 

If all those are met, then you can work through the paperwork, navigate the bureaucracy, and pay them the money for a devkit or hope you qualify for their programs allowing reduced/zero cost to use a devkit that you return.

Usually the demonstration that you can complete a commercially viable game is that you've already completed commercially viable games on other platforms, typically the PC.  After that, it is that you've got funding to hire a bunch of developers who have already completed the games.  

It often feels unfair to the people on the outside, but if you think about it objectively it makes sense from a business standpoint. They only want amazing products on their platform, although they'll tolerate good and mediocre products occasionally.

Weeeell.... Sony seems to have relaxed MANY of these constraints....

These games being released on PSN are the proof of this:

Sony seems so desperate to loose out on ANY Indie darling that they seem to flog those devkits to anyone willing to put a game out on PSN. So if there ever was a time to get a game on the Playstation as a small time Indie without much expierience and no prior game under your belt, now is the time.

I guess you still have to be lucky and probably need to lie about what your project actually is and your expierience though. Don't think Sony was actually presented with a truthful project plan for the games shown above, nor was the developer probably saying the truth about his "secure facilities" or financial situation (well, probably there wasn't much to finance anyway). Neither of these devs will probably put another game on the PSN...

EDIT: Maybe I was sounding somewhat harsh above. Well, if these games were free games distributed on the internet, developed by beginners I would say "good job! Your game compiles and is playable without errors" (which they have to be AFAIK to pass sonys certification).

These are sold for 15 bucks on a professional storefront. And from the looks of it, they are clearly not the work of a professional. So clearly, Sonys standards are not what they used to be.