Decklink Duo 2 stuttering output when adding more than 1 1080i5000 channel

Hi all,

I am trying to set up Caspar with two channels in 1080i5000, to use one channel as a video playout and the other channel as an overlay / lower third source. I have been able to set it up but the output of both channels from my Decklink Duo 2 keep having both audio and video stutters when playing back media. The only way I can get it working is to set one channel to 1080i5000 and the other one down to PAL, if I set it to anything higher the stuttering returns.

The stuttering is also visible in the diagnostics window under the decklink consumer as a short burst of late frames. It is interesting to see that the late frames also appear right after startup when there are no producers running, so it does not seem to be originating from a producer issue. The two channels also seem to be “synced” in their stutters, but this seems to start drifting when playing media.

Screenshot from the diag window and task manager after starting CCG with no producers running:

I am currently a bit lost what to do next as I tried most of the troubleshooting steps in the documentation (see below). With the setup we have (see below) should one be expecting to run 2 channels at 1080i5000? Has anyone seen this stuttering and dropped frames before on a Decklink consumer with the Decklink Duo 2?

Any help, thoughts or insights are appreciated!

Setup
OS: Microsoft Windows 10 Pro, Version 10.0.19043 Build 19043
CPU: Intel(R) Xeon(R) CPU E5-1620 0 @ 3.60GHz, 3601 Mhz, 4 Core(s), 8 Logical Processor(s)
System model: HP Z420 Workstation
Graphics card: Nvidia Quadro 2000
Boot and Media drive: SSD MTFDDAK128MAM-1J1 ATA
Decklink Card: Decklink Duo 2
CasparCG Server version: V2.3.3 LTS Stable

Logs (and config file):

Type "q" to close application.
Logging [debug] or higher severity to log/

[2022-05-01 17:19:22.644] [info]    ############################################################################
[2022-05-01 17:19:22.644] [info]    CasparCG Server is distributed by the Swedish Broadcasting Corporation (SVT)
[2022-05-01 17:19:22.644] [info]    under the GNU General Public License GPLv3 or higher.
[2022-05-01 17:19:22.644] [info]    Please see LICENSE.TXT for details.
[2022-05-01 17:19:22.644] [info]    http://www.casparcg.com/
[2022-05-01 17:19:22.644] [info]    ############################################################################
[2022-05-01 17:19:22.644] [info]    Starting CasparCG Video and Graphics Playout Server 2.3.2 4de6d18f Dev
[2022-05-01 17:19:23.470] [info]    Initializing OpenGL Device.
[2022-05-01 17:19:23.471] [info]    Initialized OpenGL 4.5.0 NVIDIA 377.83 NVIDIA Corporation
[2022-05-01 17:19:23.472] [info]    Initialized image module.
[2022-05-01 17:19:23.474] [info]    Initialized ffmpeg module.
[2022-05-01 17:19:23.474] [info]    Initialized oal module.
[2022-05-01 17:19:23.474] [info]    Initialized decklink module.
[2022-05-01 17:19:23.474] [info]    Initialized screen module.
[2022-05-01 17:19:23.474] [info]    Initialized newtek module.
[0501/171923.490:ERROR:gpu_process_transport_factory.cc(990)] Lost UI shared context.
[2022-05-01 17:19:23.502] [info]    Initialized html module.
[2022-05-01 17:19:23.502] [info]    Flash support is disabled
[2022-05-01 17:19:23.502] [info]    Initialized flash module.
[2022-05-01 17:19:23.504] [info]    Initialized bluefish module.
[2022-05-01 17:19:23.504] [info]    "C:/Users\CCG\Downloads\casparcg-server-v2.3.3-lts-stable\casparcg.config":
[2022-05-01 17:19:23.504] [info]    -----------------------------------------
[2022-05-01 17:19:23.504] [info]    <?xml version="1.0" encoding="utf-8"?>
[2022-05-01 17:19:23.504] [info]    <configuration>
[2022-05-01 17:19:23.504] [info]       <log-level>debug</log-level>
[2022-05-01 17:19:23.504] [info]       <paths>
[2022-05-01 17:19:23.504] [info]          <media-path>D:/media/</media-path>
[2022-05-01 17:19:23.504] [info]          <log-path>log/</log-path>
[2022-05-01 17:19:23.504] [info]          <data-path>data/</data-path>
[2022-05-01 17:19:23.504] [info]          <template-path>template/</template-path>
[2022-05-01 17:19:23.504] [info]          <font-path>font/</font-path>
[2022-05-01 17:19:23.504] [info]       </paths>
[2022-05-01 17:19:23.504] [info]       <lock-clear-phrase>secret</lock-clear-phrase>
[2022-05-01 17:19:23.504] [info]       <channels>
[2022-05-01 17:19:23.504] [info]          <channel>
[2022-05-01 17:19:23.504] [info]             <video-mode>1080i5000</video-mode>
[2022-05-01 17:19:23.504] [info]             <consumers>
[2022-05-01 17:19:23.504] [info]                <decklink>
[2022-05-01 17:19:23.504] [info]                   <device>1</device>
[2022-05-01 17:19:23.504] [info]                   <embedded-audio>true</embedded-audio>
[2022-05-01 17:19:23.504] [info]                   <buffer-depth>2</buffer-depth>
[2022-05-01 17:19:23.504] [info]                   <latency>normal</latency>
[2022-05-01 17:19:23.504] [info]                   <keyer>default</keyer>
[2022-05-01 17:19:23.504] [info]                </decklink>
[2022-05-01 17:19:23.504] [info]             </consumers>
[2022-05-01 17:19:23.504] [info]          </channel>
[2022-05-01 17:19:23.504] [info]          <channel>
[2022-05-01 17:19:23.504] [info]             <video-mode>1080i5000</video-mode>
[2022-05-01 17:19:23.504] [info]             <consumers>
[2022-05-01 17:19:23.504] [info]                <decklink>
[2022-05-01 17:19:23.504] [info]                   <device>2</device>
[2022-05-01 17:19:23.504] [info]                   <embedded-audio>true</embedded-audio>
[2022-05-01 17:19:23.504] [info]                   <channel-layout>stereo</channel-layout>
[2022-05-01 17:19:23.504] [info]                   <latency>normal</latency>
[2022-05-01 17:19:23.504] [info]                   <keyer>external</keyer>
[2022-05-01 17:19:23.504] [info]                   <key-only>false</key-only>
[2022-05-01 17:19:23.504] [info]                   <buffer-depth>2</buffer-depth>
[2022-05-01 17:19:23.504] [info]                   <custom-allocator>true</custom-allocator>
[2022-05-01 17:19:23.504] [info]                </decklink>
[2022-05-01 17:19:23.504] [info]             </consumers>
[2022-05-01 17:19:23.504] [info]          </channel>
[2022-05-01 17:19:23.504] [info]       </channels>
[2022-05-01 17:19:23.504] [info]       <controllers>
[2022-05-01 17:19:23.504] [info]          <tcp>
[2022-05-01 17:19:23.504] [info]             <port>5250</port>
[2022-05-01 17:19:23.504] [info]             <protocol>AMCP</protocol>
[2022-05-01 17:19:23.504] [info]          </tcp>
[2022-05-01 17:19:23.504] [info]       </controllers>
[2022-05-01 17:19:23.504] [info]       <amcp>
[2022-05-01 17:19:23.504] [info]          <media-server>
[2022-05-01 17:19:23.504] [info]             <host>localhost</host>
[2022-05-01 17:19:23.504] [info]             <port>8000</port>
[2022-05-01 17:19:23.504] [info]          </media-server>
[2022-05-01 17:19:23.504] [info]       </amcp>
[2022-05-01 17:19:23.504] [info]       <osc>
[2022-05-01 17:19:23.504] [info]          <default-port>6260</default-port>
[2022-05-01 17:19:23.504] [info]          <predefined-clients>
[2022-05-01 17:19:23.504] [info]             <predefined-client>
[2022-05-01 17:19:23.504] [info]                <address>10.105.80.107</address>
[2022-05-01 17:19:23.504] [info]                <port>6251</port>
[2022-05-01 17:19:23.504] [info]             </predefined-client>
[2022-05-01 17:19:23.504] [info]          </predefined-clients>
[2022-05-01 17:19:23.504] [info]       </osc>
[2022-05-01 17:19:23.504] [info]    </configuration>
[2022-05-01 17:19:23.504] [info]    -----------------------------------------
[2022-05-01 17:19:23.526] [info]    Initialized OpenGL Accelerated GPU Image Mixer for channel 1
[2022-05-01 17:19:23.526] [info]    video_channel[1|1080i5000] Successfully Initialized.
[2022-05-01 17:19:23.527] [info]    Initialized OpenGL Accelerated GPU Image Mixer for channel 2
[2022-05-01 17:19:23.527] [info]    video_channel[2|1080i5000] Successfully Initialized.
[2022-05-01 17:19:23.544] [info]    DeckLink Duo 2 [1-1|1080i5000] Enabled embedded-audio.
[2022-05-01 17:19:23.544] [info]    DeckLink Duo 2 [1-1|1080i5000] Disabled low-latency mode.
[2022-05-01 17:19:23.544] [error]   DeckLink Duo 2 [1-1|1080i5000] Failed to enable external keyer.
[2022-05-01 17:19:23.548] [info]    DeckLink Duo 2 [1-1|1080i5000] Initialized.
[2022-05-01 17:19:23.572] [info]    DeckLink Duo 2 [2-2|1080i5000] Enabled embedded-audio.
[2022-05-01 17:19:23.572] [info]    DeckLink Duo 2 [2-2|1080i5000] Disabled low-latency mode.
[2022-05-01 17:19:23.572] [info]    DeckLink Duo 2 [2-2|1080i5000] Enabled external keyer.
[2022-05-01 17:19:23.576] [info]    DeckLink Duo 2 [2-2|1080i5000] Initialized.
[2022-05-01 17:19:23.576] [info]    Initialized channels.
[2022-05-01 17:19:23.577] [info]    Initialized controllers.
[2022-05-01 17:19:23.577] [info]    Initialized osc.
[2022-05-01 17:19:23.637] [info]    DeckLink Duo 2 [1-1|1080i5000] Reference signal: not detected.
[2022-05-01 17:19:23.692] [info]    DeckLink Duo 2 [2-2|1080i5000] Reference signal: not detected.
diag
[2022-05-01 17:19:28.666] [info]    Received message from Console: diag\r\n
#202 DIAG OK
[2022-05-01 17:19:28.667] [debug]   Executing command: DIAG
[2022-05-01 17:19:28.667] [debug]   Executed command (0s): DIAG

Troubleshooting tried so far

  • Check the task manager after CCG Server startup or during the playout of a video. None of the resources (Disk, CPU, Graphics card) seem to be maxed out (see screenshot above).
  • Blackmagic disk speed test which returned normal results.
  • We tried to hook up a 1080i5000 sync generator to the Decklink Card. Caspar logs showed that it locked to the sync signal but this did not fix the issue.
  • We have tried a few different CasparCG Server versions: V2.0.7, V2-2-0-stable and V2.3.3 LTS Stable. All show the same issue.
  • We updated the Blacmagic Desktop video application to the latest version, which is from last April. Did not fix the issue.
  • We tried a few different source videos to play with different codecs which did not seem to fix the issue.
  • Tinkering with the buffer, latency and other components of the Decklink consumer of the config, which did seem to make time in between the stutters longer, it still stuttered every 5 to 10 seconds so it did not solve the issue.
1 Like

I would try to use a dedicated SSD for the media. Maybe that helps.

Given you report “clean” output when you create a 1080i50 and a PAL output from CasparCG, I wonder if you have a PCIe lanes issue. Blackmagic state the Duo 2 requires a Generation 2 PCIe 4 lane connection. Worth checking if your card is in such a slot, or better. Some PCs have the unpleasant issue that the lanes available can be reduced if other cards are present in defined slots. There may also be PCIe sharing happening that is using transfer capacity, for example the CasparCG producer buffer has to be sent to the graphics card, and the mixer output returned from the graphics card to the consumer buffer.

If you have access to some 1080p50 footage, H664 encoded should be fine for this test, and a suitable monitor or SDI to HDMI converter you can do a simple speed test using the Blackmagic Media Express tool. It outputs YCbCr data to the card, where CasparCG outputs (8-bit) RGBA. So one 1080p50 YCbCr signal needs around the same data transfer capacity as CasparCG operating at 1080i50. You can run multiple copies of Media Express, thus simulating a transfer rate similar to your target outputs. If the dual Media Express outputs do not operate cleanly, there is a data bandwidth issue.

I also agree with @didikunz that the SSD you are currently using could be marginal in transfer rates with system tasks and CasparCG media being accessed on the same drive.

2 Likes

That is a good one. There is a tool called SiSoftware Sandra, that can show this.

3 Likes

There is a useful piece of software called LatencyMon which I’ve used in the past to help see what is causing microstutters.

You will be able to see what is freezing during those stutters.

Hi all,

Thanks for the quick and detailed replies. those are some great suggestions!

@didikunz Great idea, I am in the process of getting a separate SSD to verify, will let you know if that helps.

@andyw Gotcha, I quickly checked the generation of the PCIe lane the decklink card is in and that should be all good, but I will still test with the BM Media Express tool and use the SiSoftware Sandra to verify if the PCIe lanes are to blame.

@frozen That looks like a great tool, thank you for that! I’ll give that a go and see what comes out of that.

I’ll report back when I have any new findings. Thanks!

@andyw you were right!

With the software recommendation of @didikunz We were able to determine that it was only using 1 of the 4 PCIe lanes!

We moved the card down a slot and it is now showing up as using 4 of the 4 lanes. The Caspar diagnostic screen also doesn’t show any late frames anymore and is now playing back two sources at 1080i5000 flawlessly.

Thanks all for the support :+1:

I also had some of the same issues. For me it turned out to be an old HDD that was not able to keep up with CasparCG reads from the Media folder. Swapped the HDD for an Samsung 860Pro SSD and that solved the issues.

Thanks for the heads up @didikunz !