Decklink consumer constant late frames, dropouts and buffering

General
, , ,
1

Hello,
new to CasparCG, trying to set it up on my computer, but having nothing but issues with the decklink card. I have downloaded the latest 2.3.3 version server, and latest client. All works fine, until decklink is added to the consumer list.

What I do to reproduce issue, is I have some random MP4 1080p video file, that I just put into the rundown and let it play. Screen consumer only means it plays fine, no issues or drops, adding decklink consumer causes heavy periodic frame and audio dropouts.
Putting just a simple static image into the rundown also leads to the same issues.

The config of the decklink is below:

<decklink>
  <device>1</device>
  <embedded-audio>true</embedded-audio>
  <latency>normal</latency>
  <keyer>external</keyer>
  <key-only>false</key-only>
  <buffer-depth>3</buffer-depth>
</decklink>

The diag command shows lots of late frames to the decklink, and audio buffering issues.
IMAGE 1

Second screenshot is comparison of running just screen consumer (right) and just decklink (left). Screen consumer only, there are no dropouts, no stuttering, all seems good.

diag2
diag21030Ă—586 315 KB

HW conf of the PC: Asrock B560M-ITX motherboard, CPU intel i3-10100T with intel UHD graphics 630. Running Windows 10 Pro, 21H2. GPU drivers up to date.
I admit, the PC is no spaceship, but it should be plenty powerful, as I can utilize all in/outs of the decklink for example in OBS, and have no dropouts or issues. I have also tested the fill+key output from OBS, also no issue there. CPU was running under 20%, GPU loaded about 50%.

There certainly is no issue with PCIe not having enough capacity, as mentioned above, for example in OBS the issue is not present and PC has plenty power to spare. Also using HWINFO64 tool, i confirmed, decklink is really using all x4 PCIe lanes. Decklink is with the latest software package 12.4.2.

Have also tried different CCG server (and client) versions. All versions since 2.1.x have this decklink issue, all 2.0.8 or older can play fine, but I am getting heaps of OpenGL errors (opengl error 1282 62abd14c), and video with a green tint, for example like this:
GREEN TINT IMAGE
The OpenGL errors according to my quick investigation seems to be cause by some deprecated opengl shader API commands being used, that are no longer supported. (maybe, not sure).

When server runs with the decklink and screen consumer, CPU load is moderate (around 35%), with just decklink it is about 20%. GPU at around 30%. Does not seem to me like a lack of power, with OBS the load is very similar, but decklink has no dropouts or stutter issues.

Have also played with the <buffer-depth>3</buffer-depth> config, deliberately putting there some high value like 32 means only the period of the dropouts gets longer and there are long parts where there is no audio or frame output.

Also noticed, that when the server channel is empty or cleared (F12), decklink buffering issues are not present. The screenshot below shows just that. First third is playing video (you can see some audio volume), then static image, then channel cleared.
IMAGE 3

Thank you for any suggestions or help, i am kind of out of ideas what to try next. Spent already two full days trying to debug this myself, without luck.

PS: I am sorry, forum does not let me upload more images, left only the one telling the most.

//EDIT: Have also tried the <custom-allocator>true</custom-allocator> setting, and in combination with “latency low”. It only made the dropout periods shorter.
//EDIT2: Out of curiosity, have tried lowering output framerate to 30. No dropouts or decklink issues. Interlaced 50 also works. But why progressive 50 (or 60) does not?

2

Casparcg needs nvidea graphics card.

3

Please elaborate on your answer. What does have GPU manufacturer in common with ability of CCG to copy rendered data to decklink?

Or was your answer related to OpenGL errors in the old CCG versions? Seems fixed in the current CCG releases, but other problems introduced, making trouble with the decklink.

What other means or tools can I use for debugging the issue deeper? Would for example WPA (Windows Performance Analyzer) help finding the issue?

//EDIT: Have searched the forum and found out numerous people run CCG on intel integrated GPU. There are even mentions of this very exact intel UHD 630 GPU, and not causing issues. I hence think your answer is invalid and the GPU is not a problem.
See for example thread here (forum user hreinnbeck Leader: using the CCG on even older Intel HD 4600, or used balte running this very exact UHD630 with 2 Fill+Key outputs: Onboard GPU - #6 by hreinnbeck
Or here is mentioned exactly the UHD630 should be working just fine: Intel NUC 9 Extreme
So I may probably report this as a CCG bug?

4

Officially only NVIDIA gpus are recommended, but if usage is simple enough then Intel can be fine. I do some of the development on a laptop with Intel gpu and haven’t had any issues with it.

I can’t think why you would b shaving this issue. Those graphs show something is taking too long but aren’t helping identifying what the cause is.

It does look in that first screenshot like you have the keyer set to external_separate_device. Could you try with that being external?

1 Like
5

I agree the integrated GPU is rather week, but I do not need anything complex, but am having buffering issues even showing a static image.

You are right, the screenshot I have posted is from the beginning of my CCG experiments, where I have had it incorrectly set to external_separate_device key out. I am using Decklink Duo 2 card, so this setting seem only to move the key output to the 2nd device within the card. I have since fixed this setting to “external”, and now I am getting the fill + key outputs at the expected connectors of the card. But switching the keyer to “external” had no influence on the buffering issue.

Is there a way I could supply some logs that would help track the issue down?

Please find below attached a recent diag screenshot, where I have deliberately set the decklink buffer to 32 long. It seems to me there are large periods of time where the card is not getting any data at all. Audio buffer runs completely dry and no frames are produced at all. Decklink buffer length seems only influences the period of the buffering drops.
! ! I will upload the image later, unfortunately I do not have access to the CCG computer right now. But I think you get the point. Setting latency low and selecting buffer depth of just 1 makes the dropouts short and fast repeating.

Since begining I have had the CCG server configured for 1080p50, but yesterday, i had an idea to try different resolution. I have tried 1080i50 (it works! :-O) and 1080p30 (also worked). Not sure if that helps any to resolve the issue. Like if the PC does not have enough throughput at the higher resolution (but weirdly, using for example OBS in RGBA mixing mode, I am able to produce key+fill, and do other image processing just fine on the same PC).

6

Since your problems only start when using a Decklink at higher than 30p, I suspect there is some kind of bottleneck in the PCIe bandwidth that you installed the card.

Usually when that’s the case, the Decklink shows drops on the diag even with an empty channel, but I had instances of the drops only appearing when playing content. I think the issue here was that the PCI bus I chose didn’t run directly to the CPU but it passed through the PCH first, so I try to avoid that.

I’d suggest to check the manual so you can confirm that your bus is actually the right bandwidth for the Duo 2 and that it runs directly to the CPU, and also that it doesn’t share its bandwidth with other onboard devices (like M.2 drives, this is getting more common now).

7

I have also seen issues with 1080p50 playback in my simple testbed system that comprises a NUC i7 (4 physical cores) using thunderbolt to connect to two Ultrastudio 4k interfaces. The NUC has an Intel UHD 655 graphics processor. This should be about double the throughput of the UHD 630 according to Intel. Using server 2.3 LTS my system works fine on 1080i25, but has audio dropouts and video dropouts when set to 1080p50.

Whilst looking at some recording and looping issues I connected the Ultrastudio interfaces to my 2015 MacBook Pro that is running Windows 10 through boot camp. The processor is also a i7 with 4 physical cores, running very slightly lower default clock (2.5 GHz where the NUC is 2.7 GHz). The MacBook Pro is running the identical CasparCG server (file copied from NUC), but has a Radeon R9 M370X graphics processor. The system works fine with 1080p50. I think it also worked at 2160p25, but my monitoring had to be done via a downconverter because my display is HD maximum.

So I agree with Sidon that it looks as if some bandwidth constraining operation is interfering with data transfer to the Decklink. Because the Intel graphics units share system memory the transfer rates tend to be lower that on the dedicated PCIe to a graphics card, so a slight delay in transfer from the GPU into a buffer could make the content late for an output frame.