CMP error on "Four Channel Off Air Logger"

aollaku · May 2, 2022, 8:51am

Hi folks,

Im using the CMP app as a record compliance system through the "Four Channel Off Air Logger" options. Every 8-10 days this application crashes. I can see the error on the log but I dont know if this issue is related to my CMP version or hardware version (Decklink driver version). Please, can you help me to investigate this error and give a suggestion if is possible? I`m very happy with this software and I want to use it for longer. Below you will find my log with crashes.

[2022-04-29 03:50:01.721] [warning] DeckLink Duo 2 [2|1080i5000] in-sync changed: -0.0142292
[2022-04-29 03:50:01.804] [warning] DeckLink Duo 2 [2|1080i5000] out-sync changed: -0.014
[2022-04-29 03:50:07.261] [info] Received message from 127.0.0.1: play 2-1 decklink 4\r\n
[2022-04-29 03:50:07.382] [info] DeckLink Duo 2 [4|1080i5000] Initialized
[2022-04-29 03:50:07.384] [info] Sent message to 127.0.0.1:202 PLAY OK\r\n
[2022-04-29 03:50:07.768] [warning] DeckLink Duo 2 [4|1080i5000] out-sync changed: 0.02
[2022-04-29 03:50:07.805] [warning] DeckLink Duo 2 [4|1080i5000] out-sync changed: 0.04
[2022-04-29 03:50:07.849] [warning] DeckLink Duo 2 [4|1080i5000] out-sync changed: 0.00116667
[2022-04-29 03:50:09.816] [info] Received message from 127.0.0.1: play 2-1 decklink 4\r\n
[2022-04-29 03:50:09.920] [error] Exception: C:\Program Files (x86)\Jenkins\workspace\casparcg-server-dep\master\src\modules\decklink\producer\decklink_producer.cpp(373): Throw in function __cdecl caspar::decklink::decklink_producer::decklink_producer(const struct caspar::core::video_format_desc &,int,const class caspar::spl::shared_ptr &,const class std::basic_string<char,struct std::char_traits,class std::allocator > &,const class std::basic_string<char,struct std::char_traits,class std::allocator > &,const class std::basic_string<wchar_t,struct std::char_traits<wchar_t>,class std::allocator<wchar_t> > &,bool)
[2022-04-29 03:50:09.920] [error] Dynamic exception type: class boost::exception_detail::clone_impl
[2022-04-29 03:50:09.920] [error] [struct boost::errinfo_api_function_ * __ptr64] = EnableVideoInput
[2022-04-29 03:50:09.920] [error] [struct caspar::tag_msg_info * __ptr64] = DeckLink Duo 2 [4|1080i5000] Could not enable video input.
[2022-04-29 03:50:09.920] [error] [struct caspar::tag_stacktrace_info * __ptr64] = 0# 0x00007FF742ACA3AE in casparcg
[2022-04-29 03:50:09.920] [error] 1# 0x00007FF742B03420 in casparcg
[2022-04-29 03:50:09.920] [error] 2# 0x00007FF742D38304 in casparcg
[2022-04-29 03:50:09.920] [error] 3# 0x00007FF742D3C63F in casparcg
[2022-04-29 03:50:09.920] [error] 4# 0x00007FF742B1ADA8 in casparcg
[2022-04-29 03:50:09.920] [error] 5# 0x00007FF742BBD432 in casparcg
[2022-04-29 03:50:09.920] [error] 6# 0x00007FF742BB9DC0 in casparcg
[2022-04-29 03:50:09.920] [error] 7# 0x00007FF742AC50E9 in casparcg
[2022-04-29 03:50:09.920] [error] 8# configthreadlocale in ucrtbase
[2022-04-29 03:50:09.920] [error] 9# BaseThreadInitThunk in KERNEL32
[2022-04-29 03:50:09.920] [error] 10# RtlUserThreadStart in ntdll
*[2022-04-29 03:50:09.920] [error] *
*[2022-04-29 03:50:09.920] [error] *

*[2022-04-29 03:50:09.920] [error] [2022-04-29 03:50:10.040] [error] [2022-04-29 03:50:10.040] [info] [2022-04-29 03:52:02.521] [info] [2022-04-29 03:52:02.523] [info] [2022-04-29 03:52:02.523] [info] [2022-04-29 03:52:02.543] [info] *[2022-04-29 03:52:02.543] [info] [2022-04-29 03:52:02.546] [info] *[2022-04-29 03:52:02.546] [info] [2022-04-29 03:52:02.552] [info] *[2022-04-29 03:52:02.552] [info] [2022-04-29 03:52:02.563] [info] [2022-04-29 03:52:02.563] [info] [2022-04-29 03:52:02.564] [info] *[2022-04-29 03:52:02.564] [info] *[2022-04-29 03:52:02.564] [info] [2022-04-29 03:52:02.573] [2022-04-29 03:52:03.617] [info] [2022-04-29 03:52:03.618] [info] [2022-04-29 03:52:03.618] [info] [2022-04-29 03:52:03.622] [info] *[2022-04-29 03:52:03.622] [info] [2022-04-29 03:52:03.622] [info] *[2022-04-29 03:52:03.622] [info] [2022-04-29 03:52:03.628] [info] *[2022-04-29 03:52:03.628] [info] [2022-04-29 03:52:03.635] [info] [2022-04-29 03:52:03.635] [info] [2022-04-29 03:52:03.635] [info] [2022-04-29 03:52:03.635] [info] *
Turn on log level debug for stacktrace.
Sent message to 127.0.0.1:404 PLAY FAILED\r\n
Received message from 127.0.0.1: ADD 1 FILE ch1/20220429035202.mp4 -b:v 15000000 -maxrate:v 15000000 -bufsize:v 1000000 -filter:v format=pix_fmts=yuv420p -filter:a pan=stereo|c0=c0|c1=c1 -s 1920:1080\r\n
ffmpeg[ch1/20220429035202.mp4] Initialized.
Sent message to 127.0.0.1:202 ADD OK\r\n
[ffmpeg] [libx264 @ 0000020A3821C080] using SAR=1/1
*
[ffmpeg] [libx264 @ 0000020A3821C080] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
*
[ffmpeg] [libx264 @ 0000020A3821C080] profile High, level 4.2, 4:2:0, 8-bit
*
[ffmpeg] [Parsed_pan_0 @ 0000020A69D6E140] Pure channel mapping detected:
[ffmpeg] 0
[ffmpeg] 1
[ffmpeg] *
*
[warning] ffmpeg[ch1/20220429035202.mp4] Unused option s=1920:1080
Received message from 127.0.0.1: ADD 2 FILE ch2/20220429035203.mp4 -b:v 15000000 -maxrate:v 15000000 -bufsize:v 1000000 -filter:v format=pix_fmts=yuv420p -filter:a pan=stereo|c0=c0|c1=c1 -s 1920:1080\r\n
ffmpeg[ch2/20220429035203.mp4] Initialized.
Sent message to 127.0.0.1:202 ADD OK\r\n
[ffmpeg] [libx264 @ 0000020A543C9640] using SAR=1/1
*
[ffmpeg] [libx264 @ 0000020A543C9640] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
*
[ffmpeg] [libx264 @ 0000020A543C9640] profile High, level 4.2, 4:2:0, 8-bit
*
[ffmpeg] [Parsed_pan_0 @ 0000020A69D6DE40] Pure channel mapping detected:
[ffmpeg] 0
[ffmpeg] 1
[ffmpeg]

vimlesh1975 · May 2, 2022, 10:54am

Glad to hear CMP is being used.
The error shown here seems due to play 2-1 decklink 4 command second time. Because for first time decklink 4 is already playing ok, and busy; 2nd time same command is throwing error.
Try to attach log when crash is happening. And Server is crashing or cmp client?
Cmp crash report may be found in system event viewer.

aollaku · May 2, 2022, 11:37am

Hi Vilmesh,

The software is very good. I would like to have some tweaks like stream capture for example. Only the server is crashing and the errors are the same every time. So the decklink is out of sync and after a while the server crash. Have you developed any other software recently with CasparCG?

Best,

Andi

vimlesh1975 · May 2, 2022, 11:41am

I am working on React caspar client .

aollaku · May 2, 2022, 11:45am

Cool. Is possible to have another capture section for streams? I can do it on CasparCG Client but I don`t have options like clip name generated automatically or making chunks every 1800 seconds.

Best,
Andi

nickb · May 4, 2022, 8:58am

Might be worth looking at invoking ffmpeg directly Andi. This will create 1800 second chunks called chunk_000, chunk_001 etc. from an rtmp stream:

ffmpeg -i rtmp://hostname:1935/stream -c copy -segment_time 1800 -reset_timestamps 1 -f segment chunk_%03d.mp4

This is what CCG is using under the hood and in my use case I could move to commodity hardware by leaving out CCG.
HTH
Nick

nickb · May 4, 2022, 9:57am

Andi,
just spotted something in the log - out-sync changed.

I suspect Windows date time is syncing and the change in time messes with CCG/Decklink and causes the crash. I’ve experienced the same thing so go to settings->time & language->date & time and disable ‘set time automatically’.

I’ve had the same problem and after seeing what happened in the event viewer disabled it and haven’t had it crash since.
Nick

aollaku · May 4, 2022, 11:10am

Hi Nick,

Thanks. I asked Vilmesh if he can add another tab for the stream capture. So will be an easy UI for the operator if he wants to change the chunk time or stream or codec. But anyway thank you very much for the tips.

Best,

Andi

aollaku · May 4, 2022, 11:13am

Oh good. But the time should sync from time to time if not then it will be not the correct time. But if this is the only way I will do as you recomendet to do it.

Best,

Andi

vimlesh1975 · May 4, 2022, 11:32am

Playing streams in casparcg from stream player module, and recording with off air logger will do this.

aollaku · May 4, 2022, 12:24pm

Hi Vilmesh,

Thanks for replying. How can I capture the output in this way? Please, can you show me a screenshot of how to do it?

Best,

Andi

nickb · May 4, 2022, 12:57pm

It’s not a great solution but it stopped us crashing on air. We have to schedule a shutdown of Caspar once a week, then we force the time sync.

Never had a problem with 2.0x versions on Windows 7, so I don’t know if it’s Windows 10, CCG or Decklink drivers

aollaku · May 15, 2022, 10:32pm

Hello, chaps,

I had the same issue again. After 15 days of running 4Channels Offair Logger, the video drifted again and on the logs, I can see a few errors generated for the second input on the decklink. I`ve uploaded the logs maybe can give more information about this issue.

github.com

aollaku/CasparCG/blob/main/README.md

# CasparCG[2022-05-15 00:23:17.558] [info]    Received message from 127.0.0.1: REMOVE 1 FILE ch1/20220514235317.mp4\r\n
[2022-05-15 00:23:17.562] [info]    Received message from 127.0.0.1: ADD 1 FILE ch1/20220515002317.mp4 -b:v 15000000 -maxrate:v 15000000 -bufsize:v 1000000 -filter:v format=pix_fmts=yuv420p -filter:a pan=stereo|c0=c0|c1=c1 -s 1920:1080\r\n
[2022-05-15 00:23:17.562] [info]    ffmpeg[ch1/20220514235317.mp4] Uninitialized.
[2022-05-15 00:23:17.562] [info]    Sent message to 127.0.0.1:202 REMOVE OK\r\n
[2022-05-15 00:23:17.563] [info]    ffmpeg[ch1/20220515002317.mp4] Initialized.
[2022-05-15 00:23:17.563] [info]    Sent message to 127.0.0.1:202 ADD OK\r\n
[2022-05-15 00:23:17.570] [info]    [ffmpeg] [libx264 @ 000002B194999B40] using SAR=1/1
[2022-05-15 00:23:17.570] [info]    
[2022-05-15 00:23:17.570] [info]    [ffmpeg] [libx264 @ 000002B194999B40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[2022-05-15 00:23:17.570] [info]    
[2022-05-15 00:23:17.578] [info]    [ffmpeg] [libx264 @ 000002B194999B40] profile High, level 4.2, 4:2:0, 8-bit
[2022-05-15 00:23:17.578] [info]    
[2022-05-15 00:23:17.587] [info]    [ffmpeg] [Parsed_pan_0 @ 000002B1643D5600] Pure channel mapping detected:
[2022-05-15 00:23:17.587] [info]    [ffmpeg]  0
[2022-05-15 00:23:17.587] [info]    [ffmpeg]  1
[2022-05-15 00:23:17.587] [info]    [ffmpeg] 
[2022-05-15 00:23:17.587] [info]    
[2022-05-15 00:23:17.597] [warning] ffmpeg[ch1/20220515002317.mp4] Unused option s=1920:1080
[2022-05-15 00:23:17.669] [info]    [ffmpeg] [aac @ 000002B15E8763C0] Qavg: 232.814
[2022-05-15 00:23:17.669] [info]

This file has been truncated. show original

Best,

Andi