Having tested with various GPUs, there are two things to consider:
- That you GPU supports at least OpenGL 4.5 or above. If you’re going to use the MIXER commands, this is a must.
- If you need to do streaming and/or recording, I urge you to look for cards that support H.264 hardware encoding. This will relieve the CPU of such tasks, and in some cases, will allow you way more performance and flexibility in video rendering. Remember that streaming and recording always imply an encoding process.
That being said, my experience so far is:
- Avoid bottlenecking your GPU by having a competent CPU. Multiple cores help way more than higher clock speed.
- Intel Core i5 or Core i7 after second or third generation will do well, mostly because of Quick Sync Video support.
- AMD Ryzen 5 or Ryzen 7 should do pretty well too, but I don’t know if Video Coding Engine is supported in encoding tasks. At least I know OBS and Handbrake use that technology for encoding.
- If you need a cost effective solution, go for a NVIDIA GTX 1050Ti or above. This will allow you at most 2 hardware encoders simultaneously. This is enough for 1 channel being streamed and recorded at the same time.
- If you want a better solution, go for a NVIDIA Quadro P2000 or above, because it’s pretty much made for this kind of job. According to the NVENC Support Matrix, the Quadro P2000 is the cheapest card that has unrestricted simultaneous encoders that will allow you to have multiple streaming and recording tasks at the same time. I’ve personally run 6 encoding sessions at the same time and performance doesn’t take a hit.
I recently made a switch from an i7-3960X with a GTX 1050Ti to an i7-8700K with a Quadro P2000 and the performance improvement is impressive.