Recently we have encountered a sequence of freeze frames while playing back XDCAM HD 422 clips on our CCG server version 2.3.2 (stable). It resulted in couple of warning messages in CCG log:
[warning] ffmpeg 97003463.mxf [540.2800/1445.5000] Waiting for video frame…
It is just an example of message, there were many of them - I can post full log for this accident if than might help.
Video output (both screen consumer and SDI output) at the time of this problem showed freeze frame for approx. a second, then jump and another freeze. This was repeated several times and eventually disappeared. Normal, smooth playback was resumed. As we have 1+1 redundant playout controlled by third party automation result of described problem was a time shift between Main and Backup video server. The one affected by freezes was later delayed with respect to properly playing server. This tome shift disappeared after next transition.
Described problem is uncommon - happens randomly: sometimes once per day, sometimes every few days.
I wonder what his ffmpeg warning may mean… could it be symptom of insufficient performance of our disk subsystem (once again - everything works fine for 99.9% of time)? Or maybe there is too small data buffer in ffmpeg unable to compensate for any disk data stream fluctuations?
I would appreciate your comments. Please let me know if any additional information regarding HW/SW configuration might help. We run CCG on WIN10 Professional. Media drive is based on 7k2 SATA spinning drives configured as RAID5 (7 Data + 1 Parity).
That is happening randomly on both machines or only on one of them? Is there any RAID related activity happening at that time? These are HDD’s not SSD’s, right?
Thank you for immediate reply and sorry for somewhat delayed answer.
Yes, it happens randomly on both Main and Backup servers. At the time when described problem occured there were no additional RAID activities i.e. FTP media transfers etc.
In our HW configuration we have HDD based RAID media storage and separate SSD M2 system drive where CCG runs.
But some RAID controller sometimes do maintenance tasks, that could drop performance. I have no idea, how this can be checked, but there should be tools…
In case it helps, some workstations with RAID5 configs have been misbehaving lately, after almost two years of just sitting there, when we tried to use it again the playout of several channels was full stutters and jumps.
Undoing the RAID and using just one SSD solved the issue.
I wish I could blame Windows updates but it was totally disabled.
Thank you very much for your post. At the very day when you wrote your comment, I have reconfigured our servers to use SSD system drive instead RAID as playout storage. No more stutter, freezes or delays since then. It is temporary solution (proof of concept so to say) due to limited SSD capacity. But I think it proves that RAID array that on paper is more than capable in terms of R/W bandwidth, may not be able to provide sustain data stream to decoder. In IT world average performance is OK, for video we need all frames on time (and in proper sequence)…