Theoretically, yes, latency can go up, especially if you're on a bandwidth limited connection and someone else starts streaming Netflix or downloading the blockchain or whatever. You can deal with this by having an "acceptable time difference window," and when the prediction goes out of this window in either direction, make the adjustment to the local offset. To avoid adjusting every time there's a delay, you may want to only make adjustments after there's been a few seconds of delta in the same direction. Time sync over a lossy TCP connection is a messy problem .... but in the end, what matters, is that events happen in the same order on all screens!