This might be a noob question, but I was just wondering how to make an editor for a game engine. And how to compile scripts and add them to the build. Like how in unity you have an editor app running and that itself can compile and run the actual game build. Do i make an engine that has the ability to compile and run a game then make an editor that can display that game? what is the logic of running a game to test it inside the editor before building? Are there two apps? or is it one app and it just updates everything when you press run and re positions them after you stop play mode? I would appreciate a detailed answer for I have made small game engines in the past but struggled to make the UI for them. Thank you in advance.

You should seperate a plain Editor and whatever Unity is because Unity is a closed system that runs your game inside the Unity Launcher App on whatever platform you start it at. This means Unity loads it's own version of the .NET CLR or whatever "scripting" you used. At the point you are getting your game code touched for the first time, there has happen a lot that you wont even recognize under the hood of the Unity executable.

Real Editor Apps like WorldEdit in Warcraft or Starcraft games are standalone tools that consume and produce just content and toggle to "play-mode" by calling the game with certain start arguments to know what map to load.

"Compiling" scripts highly depend on the environment you write and tools you expose from your engine. Is it compiled from source you have two possible options to go for; either you give the full source code of the engine so one could compile it with a given compiler or trigger compilation by any editing tool (the Unreal way) or you go for the Unity way and compile source code to some kind of embedded runtime either you wrote for your own language or use something existing like Mono or LUA.

Often there is a build pipeline involved in game development that consists of tools called during build like compiler, linker, package creation for your assets, digital signing and verifying especially when building for closed platforms like Playstation or Switch and those pipelines are mostly coded via some kind of build file or code provider (Unreal for example rebuilds it's build tool depending on each project you open in editor to have plugins added specific build-steps in there as C# files).

My way of handling this in my engine is to have the Editor App be standalone and general. Certain engines may then add certain plugins to the editor to customize workspace and functions exposed to the user and also to control steps like trigger a new build or run the game inside the editor itself. A general interface exists that plugins have to inherit so both, plugins and host application can communicate to each other

4 hours ago, Liquifire said:

Like how in unity you have an editor app running and that itself can compile and run the actual game build.

For some game mods the preferred approach is through dynamically linked libraries.  They work extremely well as a plugin approach since you can create a bunch of well-defined functions, and individual functions can be skipped (or have a do-nothing stub) when not implemented or redirected to loaded libraries if present.  If you're using a Reflection-supporting language those hookups can be handled through those mechanisms if you prefer.

Unity spent a lot of time developing their system, but effectively it works like a plugin above.  It calls a compiler and builds your C# code into a library, then it scans the compiled library to hook up all the known functions.  They hook up the functions directly the way a plugin library does, which is also leveraged for performance reasons. That's one of the reasons why all the functions are free-standing rather than virtual: if the functions for Start(), Update(), OnDisable(), OnEnable(), etc, were all virtual they would incur a cost for every call since the virtual functions need to be called, but by hooking them up only if they exist using a plugin-style architecture they avoid the overhead of calling empty stubs.

5 hours ago, Liquifire said:

Are there two apps? or is it one app and it just updates everything when you press run and re positions them after you stop play mode?

Effectively there are two, the engine and the editor. There is more to it because they've spent over a decade making it more advanced, but we can gloss over that.

Just like your main program, the editor is also a plugin for the engine system. (Caveats that it is an extreme simplification.) There is the core engine that's got the graphics, audio, physics, networking, math, I/O, and assorted other libraries. The core engine also handles a plugin architecture that allows you to load, unload, and run executable modules dynamically. They've also got the editor plugin module which gives you your menus, your object inspector, debug logs, code to launch build tools, to run and visualize profiling and performance data, etc. 

It can be started with either the editor as the main plugin module, or it can be started with your game as the main plugin module.

Startup Option One: If you start it as the editor, the engine starts and then loads the Editor plugin; the editor plugin has the object inspectors, the title bars and menus, and all the rest of the Editor experience. The editor is what you would normally consider the game.  When you use editor commands to compile and run the game in the editor, it launches the compiler as needed, then dynamically loads your game plugin and starts running it. Your game plugin is called by the editor plugin which is hosted by the engine.

Startup Option Two: If you start it as your game, the engine starts and then loads your game plugin; instead of loading the engine plugin it goes directly to your game instead. You get access to all the UnityEngine functions because they're part of the engine. However, the Editor plugin is not loaded you don't get any of the UnityEditor functions; you don't get access to the editor visualizers, the editor events, the editor build and test tools, all because the editor plugin isn't loaded and doesn't ship with the game.

If I didn't misread, for example like in Unreal Engine 4, the runtime (the game app) has simple macro like "WITH_EDITOR" kind of macros. So I think the editor code is injected in the runtime with all things frob has explained. It can pause, it can add or remove object in the editor display and put it as data that is used on the runtime, it can re-run the game, etc.

Something like this (just one of many) in their runtime code, general main file, Launch.cpp:

...
#if WITH_EDITOR
		if (GIsEditor)
		{
			ErrorLevel = EditorInit(GEngineLoop);
		}
		else
#endif
...

Then when you publish, I believe all these links are removed. Supposedly like that in general.

Again I'm just assuming, with data-oriented design (somehow), I'd focus on syncing data on both editor and the runtime (that means they are connected indirectly). For example, you drag-n-drop a cube into the game, the runtime is waiting for the editor to write the information about the cube (in a file or memory I dunno); its location, its name, its mesh, and more. After that is written, the "write" information is then dispatched to the runtime saying "hey, there's a new cube here, can you display it for me?" ... or other alternative, the runtime with the injected editor library code would be like "Oh hey look, there's a new information from the (probably) editor! a cube in location XYZ. Let's display it!" etc.

So what I'm suggesting (again, not sure if this is how Unity works) is basically there's a bridge which is the data they share, which keeps the runtime and the editor loosely-coupled but well-connected to each other with their own job. When you save the information from the editor, all the data is stored (cooked) in the .scene file that is meant for the runtime, either editor or published app.