as I found out, the time on server is the referencing time. if you want to check for example when really player fired his gun we have to do some calculation based on server time and rtt or user ping. I this right? do we have exact rtt for every message? is this all we have to do for time synchronization in real-time games?

You can never guarantee perfect synchronisation, so you can never be 100% sure of when a client performed a certain action. All you can do is decrease the degree of error. If you are continually measuring and correcting the RTT then you can be reasonably confident about the time that the client performed the action - unless their network latency changed significantly since your last measurement, which does sometimes happen organically, and also sometimes happens artificially when it might benefit the player.

You can choose to let clients send the time of their action, and compare that to your estimate based on RTT. The server could then choose to accept the client's reported time providing it doesn't differ too much from your calculation - this will give you slightly more accurate timings for the players that aren't cheating, but gives players a small opportunity to cheat by manipulating that value when it suits them.

There is no "the time it REALLY happened."

The best you can do is agree on an order of events -- "X happened before Y."

And even that is hard when two different players talk to the same server.

The way this is typically solved is that simulation is done in time steps, typically of a fixed size. The server and players then use the same time step numbers, and any action sent is tagged as "this action happened at time step X."

You can do this with variable time step size (Unreal Engine doesn't have fixed time step, for example,) and then you'd simply send the timestamp instead of the time step number, for each event.

This also lets each side bunch events from a number of simulation steps into a single network packet. For example, you may simulate 144 times per second, but only send network packets 40 times per second; each packet will then contain the events for 3 or 4 time steps. Bunching transmissions up like this does add some additional latency to the networking, but as you typically need to work with connections with high latency anyway, it's not that bad. Sending packets 40 times a second means you delay any particular event at most 25 milliseconds; players in Massachusetts connecting to a server in Atlanta will typically see more latency than that just in transmission time, anyway.

Kylotan or hplus0603:
If you have the clients send their own timestamp, does that mean you would also do some server rewind and replay?
Or would you run the server's timeline behind the clients so that incoming timestamps have usually not occurred on the server's timeline yet?
Or maybe another way of saying that is having the clients timestamp their actions slightly in the future?

what about using ntp for synchronization? does it work for real-time games?

If you have the clients send their own timestamp, does that mean you would also do some server rewind and replay?

Some people do, some don't. There are a few options. You can, for example, run remote entities "behind" and the local entity "ahead" on the client, and send the data so it arrives "just in time" on the server. Or you can extrapolate remote entities to the client time, which means you're placing entities at "guessed" positions. Or you can display both the local player and the remote players in the "behind" time frame, while sending the commands "ahead" -- this introduces significant command delay, but makes the simulation 100% deterministic. (RTS games use this model, and hide the latency using a "Yes, Sir!" animation.)

what about using ntp for synchronization? does it work for real-time games?