CasparCG Server on Headless Linux outputting FFMPEG

Hello Caspar Community.

I have a headless server sitting in a data-center, i’ve installed CasparCG Server on to it, along with nvidia drivers, NDI tools, FFMPEG.

I’m struggling to get a coherent output from Server, either NDI or FFMPEG. It outputs, and is accepting and excecuting commands okay, but it feels like the machine is maxed out on performance, as NDI runs ±5fps and drops to black, FFMPEG outputs, but the stream is unusable, pixellation, line-tearing, and generally looks like it’s out of bandwidth.

my caspar.config consumer output for ffmpeg looks like this :

<ffmpeg> <path>udp://10.10.10.239:13013</path> <args> -codec:v h264_nvenc -preset:v p5 -profile:v main -pix_fmt:v yuv420p -b:v 4M -g:v 50 -bf:v 0 -codec:a aac -b:a 128k -ar:a 48000 -ac:a 2 -format mpegts </args> </ffmpeg>

I am taking a single NDI feed in (from a tricaster) and outputting an h264 feed with HTML graphics overlayed. The machine is not running at capacity, its got Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz (32 core) and a basic RTX A400 card with nvenc running.

Whats the best practice to output in a headless environment. I’ve always used caspar with an output card, like a decklink, or as NDI out in windows, with no issues. For some reason i’m struggling with what should be a simple in and out. I think i’m getting it wrong by using it headless.

Hello
Welcome to the forum

Running headless can definitely be done in a performant way, but it can be a bit tricky to setup.

Can you post your config and startup logs?

I have some similar servers running 16 cores across 2 sockets and quadro p4000 graphics. Those too out at 2 HD channels with deck link out and 540p streaming per channel. I believe numa nodes, pcie 3.0 and memory bandwidth are limitations for processing frames causing late frames and artifacts long before the CPU or GPU metrics get maxed out. If you do figure this out please post the config and specs. I am not running headless though.

I’m running:

Supermicro x10DRU-i+ 2x Xeon E5-2667 v4 with 32gb ddr4 on 2 sticks. I need to test 8 sticks of memory to see if that helps. I believe the GPU pcie bandwidth may be an issue.

So it seems i have an FFMPEG missmatch, and it was relying on CPU decode, but able to run GPU encode.

now to try work out which ffmpeg and caspar versions all play nice together on my ubuntu install…

The other option, and perhaps im missing it, is that i don’t have caspar setup to do a hardware decode :

Here’s my config file as it stands.

?xml version=“1.0” encoding=“utf-8”?>
configuration>
paths>
media-path>/srv/media/video/
log-path>/srv/media/log/
data-path>/srv/media/data/
template-path>/srv/media/templates/
/paths>
producers>
producer id=“1”>“ndi://R240-TC/MIX 2”
/producers>
lock-clear-phrase>secret
ndi>
auto-load>false
/ndi>
channels>
channel>
video-mode>1080p5000
channel-layout>stereo
consumers>
ndi>
name>CasparA
/ndi>
/consumers>
/channel>
/channels>
controllers>
tcp>
port>5250
protocol>AMCP
/tcp>
/controllers>
/configuration>

Is there a way of forcing CasparCG Server to use nvdec for incoming ndi and mp4 streams/files?

Hardware decode still requires a copy to the CPU system memory as all consumers and producers connect via system memory. Niklas is planning to implement a vulkan based hardware decode that would keep it in the GPU, but current code doesn’t allow for that for structural reasons.

I have experienced with custom builds of casparcg that use hardware decode using nvenc, qsv, or va-api but that doesn’t solve the bottle neck that is pcie 3.0 and memory bandwidth running that CPU with only 2 sticks of memory. I have a bunch of extra memory in my nas that I need to test to see if 1 stick per memory channel helps at all. Dual socket systems with that CPU can decode 4 channels at 1080i5994 but latency and jitter in how much time it takes to render combined with memory and pcie bandwidths mean only one channel can decode without error and sometimes two.

If you figure out what works better let me know. As far as ffmpeg versions go it depends on what release or version of casparcg you have.