Best way of syncing media on two servers? Failover of servers?


#1

Does anyone have a recommendation for a reliable way/tool of syncing the media folders of two servers (like main and backup)?
And the second question: How do you realize the failover of two servers? Putting both on a vision mixer and do the failoverswitch manually or are there ways to automate this?


#2

Windows or linux?
Do you have a file server or just two video servers?
Part (A) is getting both systems to see the same data storage location, be it on a file server or by cross-mapping each other’s drives. If they can both see a file server and /media is mounted from there, you’re done :wink:.

If no file server and with cross-mapped dir’s, Part (B) is using something like rsync or freefilesync to make the folders identical. Schedule with cron (linux) or the windows task scheduler; Run the sync from both servers but at different times. Make sure that the servers’ clocks are together.

WRT failover, the first issue how you know that one has failed- both could be sending good still video even caspar has crashed (depends on how the decklink is configured) or both sending good video but one has the wrong content).

For switching, the simplest is to use two vision mixer inputs and just select the right one :grin:.

z!


#3

On linux, you could use drbd to make sure both servers are seeing the same data. While rsync will run as a program that tries to sync both after the underlaying system has written the data, drbd is part of the the system an will write the same data to both disks.


#4

Two Windows Servers, no external storage. Of course they already see each other. :wink:


#5

Use rsync (or similar) to sync from an external storage location (eg. linux fileserver) to both CasparCG servers. Never store anything manualy on the local servers!
For Main/Backup-Failover-Switching you can use a SDI router in front of the vision mixer inputs with a salvo (or macro) to route all signals to/from proper server.

Then send all AMCP commands to both servers.

Automatic failover may be triggered from by AMCP ping check to indiviual servers. If it does not respond fire the failover macro on the router.


#6

Why shouldn´t I store media on the local machines? That´s total nonsense for event machines to use an external storage…
Two machines, syncing each other… that´s it. If will give Synkron an try, as @didikunz mentioned in another thread.


#7

@TheYouth, you could try Syncthing, an open source project that works on Windows and Mac platforms with plugins available for QNAP and Synology NAS. https://syncthing.net/


#8

I agree. Server machines usually have SSD RAID to make shure that there will be no bottleneck. Getting content from external source can have impact on performance in content heavy situations.

Sync both servers, do not use external fileserver.

BTW: how you gonna handle file server crash ? :slight_smile:


#9

(sorry, this got long)
Well… IME file server machines seldom have SSD raid because they don’t need it, they use SSDs for L2 cache and spinning drives for storage, but then in my world a small file server will have at least 20TB of usable space (7 4TB drives). Pulling data from such a server at speed is not a problem although you need 10G ethernet (I’ve run the tests). I’m referring to enterprise servers, not just slamming a RAID card into a desktop PC.

Playout servers only need a single or mirrored SSDs (don’t bother with RAID5/6, with SSDs it only adds overhead). And I wouldn’t use a RAID card, I’d use the OS’s mirroring so it’s not dependent on that particular RAID card.

File server crashes… if you’re worried about that, use a redundant file server or periodically sync the workstations from the file server. Let the server do what it designed to and the playout/CG systems do what they’re designed to. Again IME quality file servers are some of the most resilient pieces of software out there excepting carrier switches and medical devices (I routinely see 180 days uptime before restarting to apply updates).

z!


#10

i’m talking about network bottleneck. you need serious money to build reliable enterprise grade file server plus network gear to keep up with it. it may have sense on broadcast station, where you setup everything once and usually do not touch for years, content files are mostly the same for months etc. but on event world when you put data on the fly, use once and forget, why just not to use 2xSSD RAID0 on media servers and sync them using script or 3rd party software?


#11

Forget the network bottleneck. Put 10G cards in it and you‘re absolutly fine on the network side.

In my case I just need a redundancy for events, not an TV Station where the File Server makes absolutly sense.
I will try different 3rd party tools for syncing :slight_smile:

Will post the results then…