What server specs are your machine running? I have noticed the same behavior that VLC plays it flawlessly and the server stutters on the same file. I also solved it by encoding the video to a lower bitrate. Probably to do with ffmpeg being what it is and casparcg not really being a video playout server.
Best codec for video playback - General - CasparCG Community Forum (casparcgforum.org)
Looking at the link you could try quicktime h.264.
Have you tried with .mp4 instead of .mov?
Does your atem have some delay in its settings? Probably not since it dropped from 4 seconds to 1 second but worth a shot.