Timestamp Drifting In Live Capture Situations

Some video capture devices have a timestamp drifting problem after running for a while. When this problem occurs, audio and video streams in the compressed stream might be synchronized at the beginning of the capture but will start to lose synchronization after a couple of hours.

In our tests, some Decklink video capture devices would start to show this problem after capturing video for 8-24 hours. The drift is usually gradual, noticeable only after running for a few hours.

There is another problem in which the audio and video are slightly off right from beginning. There are some capture devices in which audio and video are off by about 60-90ms. In such cases, the lack of synchronization is very small and hard to detect unless inspected very closely.

The LEAD H264 Encoder, LEAD H265 Encoder and LEAD AAC Encoder compressors can automatically correct both of these problems. Enable this timestamp correction by setting the ILMH264Encoder::EnableDriftCorrector, ILMH265Encoder::EnableDriftCorrector and ILMAACEncoder::EnableDriftCorrector properties to VARIANT_TRUE.

This timestamp correction should be enabled only if you detect this problem and only if you are in a live capture situation. Do not enable this mode in a file conversion situation.

In a live capture, the encoder expects the audio and video samples to arrive at the encoder at precise intervals. So if the samples deviate from that, it is an indication of a problem (and LEAD encoders can correct this).

In certain situations, the capture device gets very confused and the difference in timestamps for the audio and video streams becomes quite large (over 500ms). The encoders are unable to correct this problem and we noticed the audio and video are still unsynchronized even after the drift correction. The only way to correct this problem is to restart the capture graph. The encoders will send the EC_STREAM_ERROR_STILLPLAYING (0x07) DirectShow event with Param1 set to LTMM_E_STREAMS_ARE_TOO_FAR_APART (0x80050062) and Param2 set to a FOURCC identifying which encoder sent the notification. The application should listen for this event and restart the capture when this happens.

But in a file conversion, the samples can arrive much faster without that being a problem. Setting the LEAD encoders to make the corrections in this case would cause incorrect results.

The application developer knows whether you are in a live capture and whether the drift correction is needed. It is the responsibility of the developer to enable or disable the drift correction mode by setting the EnableDriftCorrector property to VARIANT_TRUE (or True in .NET).