Distributed Transcoding

The LEADTOOLS Elementary Stream Source and Writer Filters allow developers to create applications that use multiple physical machines to transcode a single file, reducing the total conversion time. The process of splitting the source file into multiple pieces, transcoding each of those pieces on a separate machine, and merging those transcoded files back into a single file is done seamlessly without any gaps at the split points or loss of audio/video synchronization.

The filters are included as part of the LEADTOOLS Multimedia Suite.

Perform the following steps to create a multi-machine transcoding application:

Creating the split files

Create a convert control to split the source file into smaller files. These files will be transcoded on separate machines.
Specify the ConvertCtrl.SourceFile for the conversion.
Specify the TargetFormat of the Convert Control to the LTES format.
Obtain an instance of the LEAD Elementary Stream Writer filter using the ConvertCtrl.GetSubObject method and set the following properties:
- VideoClipMethod - This will tell the Writer whether to split the source into a fixed number of files, or base the number of output files on a specified duration of each file.
- VideoClipDuration - If splitting the source into a variable number of output files, set this to the desired duration of each split file.
- VideoClipCount - If splitting the source into a fixed number of output files, set this to the number of split files the writer should create.
- TargetFormat - Set this to the final format to be used when the split files are merged back together.
Set the TargetFile to .ltesc. By specifying LTESC as the extension, you are telling the writer to split the source, and create a catalog file (explained below).
Start the conversion. When complete, you should have several split files (determined by the settings in the Writer Filter), and a catalog file.

Transcoding each split file

In a typical scenario, each split file would be copied to a separate machine and the file would be transcoded as described below:

Create a convert control to transcode the specified split file.
Specify the ConvertCtrl.SourceFile for the conversion. In this case, it would be the corresponding split file.
Add the desired video compressor to the convert control. The same compressor must be used on each split file.
Specify the TargetFormat of the Convert Control to the LTES.
Obtain an instance of the LEAD Elementary Stream Writer filter using the ConvertCtrl.GetSubObject method and set the VideoClipMethod to eLTESWriter_ClipMethod_None. Since the original file has been split, it is unnecessary to split it again. We are only performing a normal (1:1) conversion here.
Set the TargetFile to .ltes. Specifying LTES as the extension, tells the writer to create single output file (do not split the source).
Start the conversion. When the conversion on each machine is complete, you will be ready to merge all the individual clips back together.

The above process should be repeated for the audio stream as well. See Notes on stream types below for more information.

Merging the split files

Once all split files have been transcoded, you will be ready to merge all the individual clips back together. The source path of all transcoded clips (audio and video) should match the paths of the corresponding clips in the original catalog file:

Create a convert control to transcode the specified split file.
Specify the ConvertCtrl.SourceFile for the conversion. In this case, it would be the catalog file. When using a catalog file as the source of a conversion, the output will be the merged output of all clips in the catalog file.
Specify the TargetFormat of the Convert Control to the desired output. This should match the TargetFormat set earlier in the writer filter.
Set the TargetFile to the desired output file name.
Start the conversion. When the conversion on each machine is complete, the result will be a single file that was compressed using multiple machines.

The above process should be repeated for the audio stream as well. See Notes on stream types below for more information.

LTESC Format

The LTESC format (LEADTOOLS Elementary Stream Catalog) is used as an index file and contains XML which represents all information necessary to merge all split files back together. Below is an example of a LTESC file.

<?xml version="1.0" encoding="utf-8" ?> 
<ltesCatalog> 
  <stream id="stream 1" type="{73646976-0000-0010-8000-00AA00389B71}" > 
      <clip src="test-1-c1.ltes" start="0" duration="100100001" /> 
      <clip src="test-1-c2.ltes" start="0" duration="100100111" /> 
      <clip src="test-1-c3.ltes" start="0" duration="100100111" /> 
      <clip src="test-1-c4.ltes" start="0" duration="100100111" /> 
      <clip src="test-1-c5.ltes" start="0" duration="20020000" /> 
  </stream> 
  <stream id="stream 2" type="{73647561-0000-0010-8000-00AA00389B71}" > 
      <clip src="test-2.ltes" start="0" duration="419635001" /> 
  </stream> 
</ltesCatalog>

Stream - This represents a stream in the catalog file. Since audio streams usually convert very quickly, the audio stream is never split. Only the video stream is split.

Type - Represents the Guid of the stream major type. In most cases, this is audio or video but other stream types are supported as well. As stated above, only video streams will be split.

Clip - Each stream contains a collection of clips. These are the individual files which were split from the source file, transcoded, then merged back together.

Src - The file path for the corresponding clip.

Start - The start time for the corresponding clip.

Duration - The length of the corresponding clip.

Notes on stream types

When merging the split files back together, it is important that the path to all transcoded split files is updated in the catalog file. You can either update the catalog file directly, or create a separate catalog file which is a copy of the original, but with updated file paths.

The transcoding process for the video stream is the most CPU-intensive part of transcoding and it is responsible for the majority of the transcoding time. The transcoding process for an audio stream is much faster and less CPU intensive. As a result, the LTES writer only splits the video stream. Any other stream (like the audio), can still be transcoded, but the entire stream will be transcoded at once.

Known issues

The Microsoft MPEG4 decoder will occasionally drop the last frame of a conversion. Synchronization will not be lost in the merged video, but the last frame of each split file will be lost. To resolve this issue, we recommend using the LEAD MPEG4 Decoder. You can do this by either increasing the merit of the LEAD Decoder, or setting the SetPreferredFilter Property to the LEAD MPEG4 Decoder.

Distributed Transcoding

See Also