CasparCG crashes randomly on Ubuntu 24.04

General
1

Hi!

I’m having this issue with CasparCG crashing randomly with illegal instructions error.

/home/casparcg/run.sh: line 12: 2493 Illegal instruction (core dumped) LD_LIBRARY_PATH=“$BASE_DIR” “$BASE_DIR/casparcg-server-2.5” “$BASE_DIR/config/casparcg.config”

CasparCG crashed with code 132 — restarting in 2s…

Type “q” to close application.

Logging [info] or higher severity to log/

I tried many different approaches, fixes and builds and nothing seems to work. It happens on two different machines.
Does anyone have any idea what it is and how can i fix this?

2

How have you built this? have you tried with the deb packages?

Maybe your cpu doesnt support AVX2? The 2.5.0 release was built requiring that, but that has been stepped back in the not yet released 2.5.1 (but the beta builds of 2.5 on the website since then are updated)

3

I started with official deb package.
My cpu does support avx2. Checked it a couple times.
I went into a rabbit hole of trying to rule out its cef issues.
Tried also a build from source with the portAudio from GitHub - gmeisel01/server at portaudio-consumer · GitHub
The second machine is an older one so it could be missing avx2, but the one i’m using now is quite modern with i7-14700KF.

Now im testing on Ubuntu 26.04

Will try beta builds next

4

I can confirm now the crashes happen because of CEF. I did a fresh Ubuntu install, installed deb package of beta build 2.5.0 f517f48 Dev. Had it running all night with just videos, then in the morning i added some templates and in a couple minutes it crashed.
Does anyone have an idea what can i do to fix this?

5

I’m guessing your templates are using quite a lot of memory? It’s probably the same issue as this:

The tl:dr; seems to be to “build CEF without sysroot” (in an environment with glibc >= 2.33). …which is rather inconvenient for regular users of casparcg… I know of others that have done just that, and it resolved the issue for them. I’ll see if I can get hold of their cef build and how to make that available for you to test

6

Yes, that is exactly it. If you manage to get a build, that would be great.

I’ll get back to it next week.

The templates I use are mostly borders for PIP layers, will also try to check how memory hungry they are.

7

@Julusian How would we go about doing this? I can upload the custom built cef archive (it’s a drop in replacement for the “official” releases that we use, exact same version) to the dependencies releases. I’m not 100% sure what the consequences are if you try to run this build on a system with a glibc version earlier that 2.33, but from what I can tell, even ubuntu 22.04 came with a newer glibc than that.

Should I just replace the cef archive that we have, or should we provide som kind of “alternative cef” that you opt-in for via cmake options? I think it could be fine to try to just replace it. Ppl running the dev builds know what they are in for :slight_smile:

8

I’ve managed to build cef without sys root and it 100% stopped the crashes.

I have the same question as one of the commenters from the GitHub thread you linked. How is this not more widespread?

9

I am not opposed to replacing the default CEF build, it would keep variations in build config to a minimum. I don’t think the glibc change is a problem, as long as it is compatible with our target ubuntu releases.

I think that before we change it we need to make sure that the process of building this is documented (relative to the official building docs), so that I know how to rebuild CEF in the same way when we want to update it (and technically its needed for GPL reasons)

10

The archive I have has been built in a docker container. It would be easy to make a minimal Dockerfile that we can include or link from documentation

11

@mko1989 Hi, do you have a way to share the cef build with the community?
Having the exact same problem for some time

12

It’s on my GitHub in releases, but it was built against a custom CasparCG build with port audio and some other enhancements. It should still work tho.

13

FWIW there will be an official Debian package for CEF at some point:

In the meantime, unofficial debian packages can be picked up on the Releases page.

14

Hi all,

I think I’m hitting this same crash on two separate machines running casparcg-server 2.5.0 stable on Ubuntu 24.04 LTS. We’re getting multiple of these crashes per day.

The crashes occur when using HTML templates (our template sizes are around 5-20KB) and we only ever have at most two HTML templates on at once.

casparcg-server logs show normal activity before the crashes, and then nothing else during or after each crash.

The only useful log I can find is inside the Kernel Journal (using journalctl -k) which occurs at exactly the same time as each crash:

15:52:14  kernel: traps: MemoryInfra[14141] trap invalid opcode ip:713c1e7270aa sp:713baa7fa730 error:0 in libcef.so[77260aa,713c1984a000+b16d000]

Here’s the log of casparcg-server exiting at the same time (using journalctl -u casparcg-server)

15:52:15  systemd: casparcg-server.service: Main process exited, code=dumped, status=4/ILL

Something else interesting, the byte offset (77260aa) is the same on every crash, on both machines. Thought I’d mention that in case it points to somewhere interesting in libcef?

This is quite a systemic bug for us currently! Let me know if you need any other logs or information.

Best,

Josh

15

Looks exactly the same.
Did you try the CEF from this link

or from my build:

16

Hey,

I’ve tried your build and my own, both no-sysroot, CEF 142. Caspar hangs when initialising CEF and never launches a subprocess.

Both hang at the same point:

[info] Initialized artnet module.
[info] [html] Using CEF cache path: /opt/casparcg/cef-cache

Then it deadlocks and hangs forever, the HTML module never initalises like it normally does.

I swapped the whole set (everything from the build’s Release and Resources folders swapping files within /usr/lib/casparcg-cef-142) so I’m guessing I’m missing something else?

As you say, I’m surprised this issue is not more widespread.

Thanks,

Josh

17

This looks like it maybe a permission/ownership issue, where cef fails to write to cache.

in my setup i moved everything to home/casparcg/* to avoid those issues. I also strictly call lib dir with libcef in my run.sh

set -f

CASPAR_ROOT=“${CASPAR_ROOT:-/home/casparcg/highascg}”

CASPAR_LIB=“${CASPAR_LIB:-$CASPAR_ROOT/lib}”

export LD_LIBRARY_PATH=“$CASPAR_LIB”

unset LD_PRELOAD

export DISPLAY=“${DISPLAY:-:0}”

export XAUTHORITY=“${XAUTHORITY:-/home/casparcg/.Xauthority}”

CONFIG_PATH=“${CASPAR_CONFIG:-${CASPAR_CONFIG_PATH:-$CASPAR_ROOT/config/casparcg.config}}”

CASPAR_BIN=“${CASPAR_BIN:-$CASPAR_ROOT/bin/casparcg}”

exec 9>>“${CASPAR_RUNSH_LOCK:-/tmp/caspar-runsh.lock}”

flock -n 9 || exit 0

18

Hey,

Tried this a couple of times today and yesterday, it still seems to deadlock for me? I’ve tried running it as a different user from home too.

@mko1989 were you running this with your CasparCG fork, or the standard v2.5.0 build?

@Julusian would this crash issue be something we need to raise on GitHub? It’s preventing us from going live with Caspar at the moment.

Any logs or information you need, feel free to let me know.

Thanks,

Josh

19

i am running this cef with many different builds of caspar on ubuntu 24 without issues for months now.

including the stable 2.5.0

20

Hey @mko1989

Thanks so much for your help over the last week, I’ve solved the deadlock, it was caused by a different issue with an NVIDIA driver. Was thinking it was caused by the build due to how close it was on boot with the sysroot issues.

The no-sysroot build works for me now, and I’ve run graphics continuously with no issues!

Thanks so much,

Josh