Have you tried playing only in the first channel and routing it to the others?
If playback is OK there, you could try and tweak the FFmpeg threads setting in the config file. What’s the GPU usage when the problem happen?
If you still have problems, you can always use 4 running servers side by side, each one with two outputs.
Also double check the codec. If you use intra-frame compressed codecs, you might find some performance drops. I’ve had much better results using H.264. This issue was addressed partially in this post, and here is an example of that DeckLink working with six outputs from a single server instance (another one was outputting fill+key from the same card)