You'll probably want to use a real-time clock rather than a wall-clock clock for that reason.
Linux: clock_gettime(CLOCK_MONOTONIC_RAW)
Windows: QueryPerformanceCounter()
OP is using Java, System.nanoTime() provides this functionality (as of JDK8, implementation uses: QPC on Windows, CLOCK_MONOTONIC on Linux, mach_absolute_time on OS X and iOS).
So how do I send server's timestamp without losing accuracy due to latency?
It's impossible to avoid inaccuracy from latency. You could try to compensate for it, but depending on your needs, it might be a waste of effort. Your focus should be on making sure things are happening in the right order, objects are moving at the right rate, and that all of this is happening within an acceptable time difference between clients.