Multi-channel nvenc

paveltonev · January 4, 2024, 5:18pm

Hi,

We would like to use GPU for multiple stream encoding in CasparCG server, but cannot start more than one stream.

The server is started with <log-level>trace</log-level>

Scenario1: The server is started without any defined channels and execute:

ADD 1 STREAM udp://192.168.0.24:9876?pkt_size=1316&bitrate=8000000&overrun_nonfatal=1&fifo_size=300000&buffer_size=130000 -format mpegts -codec:v h264_nvenc -codec:a aac

The result of this command is #202 ADD OK , i.e. the stream is added successfully and everything works as expected.

ADD 2 STREAM udp://192.168.0.24:10876?pkt_size=1316&bitrate=8000000&overrun_nonfatal=1&fifo_size=300000&buffer_size=130000 -format mpegts -codec:v h264_nvenc -codec:a aac

The result of this command is #400 ERROR, nothing more, the stream is not added

Scenario2: Channel are defined in casparcg.config


  <channel>                                                                    
    <video-mode>1080p5000</video-mode>                               
    <consumers>                                                      
        <ffmpeg>
            <path>udp://192.168.235.94:9876?pkt_size=1316&bitrate=8000000&overrun_nonfatal=1&fifo_size=300000&buffer_size=130000<path>                                                                                           
            <args>-format mpegts -codec:v h264_nvenc -codec:a aac</args>
        </ffmpeg>                                                              
    </consumers>                                                                │
</channel>
<channel>                                                                    
    <video-mode>720p5000</video-mode>                               
    <consumers>                                                      
        <ffmpeg>
            <path>udp://192.168.235.94:10876?pkt_size=1316&bitrate=8000000&overrun_nonfatal=1&fifo_size=300000&buffer_size=130000<path>                                                                                           
            <args>-format mpegts -codec:v h264_nvenc -codec:a aac</args>
        </ffmpeg>                                                              
    </consumers>                                                                │
</channel>

Once the server is started, there is a Segmentation Fault error and the server cannot be started.

Please advice how to manage the situation.

Thank you!

didikunz · January 4, 2024, 5:22pm

What server version do you use?

paveltonev · January 4, 2024, 6:44pm

v2.4.0-beta1

paveltonev · January 6, 2024, 6:46pm

Can you help me?

Julusian · January 6, 2024, 8:44pm

I am not able to reproduce a crash for scenario 1, but that could be because I have to modify the command a bit due to running on linux with an intel gpu. It does accept both commands and appears to start both consumers.

What gpu do you have? is it able to encode multiple streams concurrently?

For 2, there are a few mistakes in your config which could be the reason for it crashing (that version does often crash instead of cleanly exiting), it was probably trying and failing to report that your config was invalid

On your </consumers> lines, there is a stray | character
After your urls, you have <path> when it should be </path>
Both of these are causing it to be invalid xml causing the parsing to fail

paveltonev · January 8, 2024, 12:57pm

Dear Julusian,

Thanks for pointing me my error in the config file. Now I am able to start multiple nvenc stream consumers, under linux on the two GPUs Quadro M4000 - 8GB ram concurrently. Now scenario2 is working, while scenario1 not yet, anyway we have a working solution.

Thank you!

anuj · January 17, 2024, 3:51pm

Hi Everyone,
I am currently facing challenges with my CasparCG server running inside a Docker container, specifically related to GPU encoding. I have encountered issues with both Scenario 1 and Scenario 2 mentioned by @paveltonev, and I’d appreciate any assistance in resolving them.

Details:

CasparCG Server Version: 2.4.0
Docker Environment: Docker image build using the tools/linux/Dockerfile
GPU: NVIDIA Corporation TU104GL [Tesla T4] | NVIDIA-SMI 535.54.03 | Driver Version: 535.54.03 | CUDA Version: 12.2
CLOUD INSTANCE : g4dn.xlarge
HOST OS: Ubuntu 22.04.3 LTS
Error Log:
[2024-01-17 09:05:18.261] [info] Received message from 0.0.0.0: ADD 1 STREAM udp://hls_nvidia:6666 -format mpegts -codec:v h264_nvenc -codec:a aac\r\n
[2024-01-17 09:05:18.262] [info] ffmpeg[udp://hls_nvidia:6666] Initialized.
[2024-01-17 09:05:18.262] [info] Sent message to 0.0.0.0:202 ADD OK\r\n
[2024-01-17 09:05:18.266] [error] [ffmpeg] [h264_nvenc @ 0x7f685099b400] Cannot load libnvidia-encode.so.1
[2024-01-17 09:05:18.266] [error]
[2024-01-17 09:05:18.266] [error] [ffmpeg] [h264_nvenc @ 0x7f685099b400] The minimum required Nvidia driver for nvenc is (unknown) or newer

It is crashing the casparcg server but when I just use -codec:v libx264 it is using mpeg2video codec. And I need it to use GPU for encoding as well as decoding which is not happening here.

If anyone has experience with CasparCG in a Docker environment or has encountered similar issues, I would greatly appreciate your insights and suggestions.

paveltonev · January 18, 2024, 3:18pm

Hi Anuj,

If you are using CasparCG server 2.4.0-rc1, then I don’t think it is possible to use hardware encoding in the way as the Dockerfile is.

You can view here following statement:

If not using system ffmpeg, run ./_deps/ffmpeg-lib-src/ffmpeg/install-ffmpeg-dependencies to install the dependencies needed by the ffmpeg build

You can find the same code in ./tools/linux/Dockerfile

A possible solution is to build ffmpeg image which you either use as basis or copy dependencies from.

anuj · January 28, 2024, 2:05pm

hey @paveltonev thanks for the message,
it is a shame that we can not use hardware encoding in default Dockerfile, but i tried other ways like copying dependencies from another ffmpeg image and it is not working, may be I am doing something wrong there, will figure it out.
Again I appreciate your message it saved me ton if time.

paveltonev · January 29, 2024, 11:11am

Hi @anuj ,

Here are general outlines how I did it:

Installed nvidia drivers. Run “nvidia-smi” there you will see your runtime cuda version.
Build and install a custom CudaGL(OpenGL + Cuda) image from here, base on your OS and cuda version.
Use as a template from v.2.4.0-beta the file casparcg-server/tools/linux/ffmpeg/Dockerfile, where you should use the cudagl image as basis. There you can modify your ffmpeg version, the flags you build it, etc. Build and install the image.
Use as basis from v.2.4.0-rc1 the file casparcg-server/tools/linux/Dockerfile where you should take as basis the built ffmpeg image from 3. above to build the casparcg server. In second step again use ffmpeg image from 3. above, but also you must add vulkan drivers in last steps the image, enable corresponding nvidia-driver capabilities, I make it to “all” and make connection from this casparcg image to “libGLX_nvidia.so.0” library.

As a result I am able to build ffmpeg image with ffmpeg 6.1.1 ffmpeg nvcodec version 12.1.14.0 using running successfully on dual core with dual gpu.