I usually buy a 19" case and add a standard desktop (or server) motherboard to it. Does the machine reside in a server room, or do you have it close to your workplace? For the workplace I would not recommend buying a 1RU or 2RU machine, as these usually have very small fans, that create a lot of noise. 3RU or 4RU machines have bigger fans, that do not run as fast and are much quieter.
Definitely a Windows Workstation license, as there is nothing you need, that is not included with it and would rectify the higher price.
GPU: nVidia Quatro or better.
CPU: Intel i9 CPU’s are very fast. i7 may also work.
RAM: 16GB is ok, 32GB is luxurious.
SSD’s (no HDD’s any more). Use a SSD for system and software and a separate one for Caspars media and template folders. If you play a lot of videos with high bit-rates I would recommend building a RAID 0 for the media. I usually do a RAID 1 for the system drive, that keeps you on air, if you have a drive failure, but maybe that is only my paranoia.
Decklink: I think the 8K Pro should now work for multiple channels, but I never tried it myself. AFAIK the Duo 2 is not capable of doing fill & key in 1080p50.
I have a case from SLIGER, 3U so i get normal pcie slots. Also with another benefit that the case depth is acceptable.
I have a decklink duo 2 for 4 channels 1080p50. I have run key/fill at 1080p50 on it.
Then I run ubuntu desktop and have a small kvm to remote into the machine. Now @Julusian has made changes to the server so casparcg could be started headless automatically via a service. So really you could run Ubuntu-server for a headless setup. But is there a reason to run windows in your case? A kvm for remote control has proven very useful for me at least.