CasparCG server - recovery management

General
1

Hi,

With regards to the possibility for consumer and/or producer or the server to fail for some reason, can you recommend existing software that can manage CasparCG server in case of following failures:

  1. Consumer failure - to start a new consumer upon failure of existing one.
  2. Producer failure - to start a new producer from the point before the failure.
  3. CasparCG server failure - to start a new server with corresponding producers at the points before the failure.

Thank you!

1 Like
2

I am pretty sure caspar does not support restarting individual producers or consumers. The entire server would have to restart.

As far as I know there is not an existing software that “handles” the caspar server in the way you want it to. With that said the team I am a part of is currently developing a “caspar manager”. But still, not possible to restart producers and consumers.

It would be interesting if it is possible to change the config and have the server initialize a new channel or remove live. But I am not sure it the server can handle these types of features.

Interesting to read what others have to say in this topic.

Edit:
Check out Sofie Automation. They have a state machine of sorts that keeps track where Caspar was if it crashes.

3

I think it depends a lot on what you mean by ‘consumer failure’ and ‘producer failure’.
Each type of producer/consumer has different failure modes/scenarios, some of which could be solved by clearing the layer and readding the producer/consumer, some of which will require a restart of the server or potentially even the os.

For the ‘casparcg server failure’, I think that depends on the software controlling casparcg. Some existing systems detect this and are capable to restoring everything by themselves. But in a lot of cases they will not. And the steps to get it back into the correct state can require knowing what commands were issued (although the xml/osc state might now contain enough to achieve this), making this a hard problem to solve.

It would be interesting if it is possible to change the config and have the server initialize a new channel or remove live. But I am not sure it the server can handle these types of features.

I have experimented around with this, and there is no technical reason that it couldnt be done. But it definitely carries some risk of crashing, and of control issues as removing the first channel results in the ‘id’ of each channel changing, unless we also allow non-sequential channel ids.

1 Like
4

I was thinking about the use-case for a configuration type page inside a client. I understand that it could be risky to be removing and adding producers in a live production environment.

Thank you for the detailed reply.

5

Now I am driven by a concrete bugs now in the server which cause in my case consumer to fail or producer to fail.

In the consumer case a possible solution could be to restart the consumer at certain period between clips with commands:

  1. ADD 1-700 STREAM udp://…(Add initial stream with id 700)
    … Wait certain amount of time …
  2. ADD 1-800 STREAM udp://…(Create another consumer with id 800)
  3. REMOVE 1-700 (Remove the first consumer with id 700)

In the producer state and server restart, I agree that systems like Sofie could work.