DC 5.8 Digibatch configuration

Digibatch is the system used for transcoding assets into different renditions. For instance it generates thumbnails for all assets.

This guide describe some of the configuration options that are available to fine tune the performance and stability of Digibatch.

Digibatch uses the following third-party tools for doing the actual transcoding of assets

ImageMagick for Images, pdf, psd, illustrator and more.
FFMPeg for video
Aspose for office document and 3d files

Along some configuration for Digibatch the above tools can be tweaked to perform differently.

Digibatch configuration

Digibatch comes with very limited configuration options, as most of it is handled directly in the application. There is however one configuration used to limit the amount of jobs executed in parallel.

A jobchain in Digibatch consists of a set of directions which transcodes the file into different renditions and notifies the DAMCenter when it is done.

These directions can be executed in parallel. The options for parallel execution comes in three flavous

None
Limited
Unlimited

The none option is used solely for FFMPeg video transcoding as it is assumed to take up a lot of resources. The unlimited flavor allows unlimited parallel execution and is used for jobs that take up very little resources. This includes jobs like computing hashes and notification jobs.

Lastly we have the Limited option which is used by the transcoding jobs (expect video). This options allows a configurable number of jobs to run in parallel. This configuration is controlled by the Appsettings.json file of Digibatch which looks as follows

{
  "Logging": {
    "LogLevel": "Information",
    "Overrides": {
      "Microsoft.EntityFrameworkCore": "Warning"
    },
    "EnabledCentralizedLogger": true
  },
  "RabbitMQ": {
    "host": "RabbitMqHost",
    "username": "RabbitAdmin",
    "password": "RabbitPassword",
    "vhost": "RabbitVHost"
  },
  "InstanceConfiguration": {
    "InstanceName": "batch",
	"MaxParallel": 3
  }
}

Specifically the parameter MaxParallel (which may not be there by default and therefore has to be added manually) controls the Limited option and specifies how many instances can be run in parallel.

ImageMagick configuration

ImageMagick has a ton of configuration options and comes in many different flavors. It is out of the scope of this documentation to go through it all and therefore we refer to ImageMagicks own documentation.

In this section a few of the important configuration options there are

ImageMagick builds

ImageMagick comes in different versions. The things that can be adjusted are

High dynamic range imaging (HDRI)
Bit size (Q8 and Q16)

The build that is provided with the standard Digibatch installation is the one that gives the best possible image quality (Q16-HDRI).

The quality vs. performance is always a trade of and if one can live with lower quality for better performance, then a different version of ImageMagick may be installed.

ImageMagick configuration parameters

ImageMagick has a wide range of configuration parameters. A full list can be found in their documentation

The important configuration file to know is the Policy.xml located in the installation directory of ImageMagick. This file contains the following section

<policymap xmlns="">
<!--  <policy domain="resource" name="temporary-path" value="/tmp"/>  -->
<!--  <policy domain="resource" name="memory" value="2GiB"/>  -->
<!--  <policy domain="resource" name="map" value="4GiB"/>  -->
<!--  <policy domain="resource" name="width" value="10KP"/>  -->
<!--  <policy domain="resource" name="height" value="10KP"/>  -->
<!--  <policy domain="resource" name="list-length" value="128"/>  -->
<!--  <policy domain="resource" name="area" value="100MP"/>  -->
<!--  <policy domain="resource" name="disk" value="16EiB"/>  -->
<!--  <policy domain="resource" name="file" value="768"/>  -->
<!--  <policy domain="resource" name="thread" value="4"/>  -->
<!--  <policy domain="resource" name="throttle" value="0"/>  -->
<!--  <policy domain="resource" name="time" value="3600"/>  -->
<!--  <policy domain="coder" rights="none" pattern="MVG" />  -->
<!--  <policy domain="module" rights="none" pattern="{PS,PDF,XPS}" />  -->
<!--  <policy domain="delegate" rights="none" pattern="HTTPS" />  -->
<!--  <policy domain="path" rights="none" pattern="@*" />  -->
<!--  <policy domain="cache" name="memory-map" value="anonymous"/>  -->
<!--  <policy domain="cache" name="synchronize" value="True"/>  -->
<!--  <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/>  -->
<!--  <policy domain="system" name="max-memory-request" value="256MiB"/>  -->
<!--  <policy domain="system" name="shred" value="2"/>  -->
<policy xmlns="" domain="system" name="precision" value="6"/>
<!--  <policy domain="system" name="font" value="/path/to/unicode-font.ttf"/>  -->
</policymap>

Where a lot of things can be adjusted. There are a few of them that should be highlighed

temporary-path is used to control where ImageMagick stores temporary bytes while transcoding. If the memory limit is exceeded this directory is used. This must be a ssd disk.
memory controls the amount of RAM ImageMagick is allowed to use. This can be adjusted to ensure the server wont run out of memory. If the memory limit is exceeded then the files are written to the temporary path.
thread is used to control how many threads are allowed to be used while doing the transcoding
throttle throttles the transcode. Essentially it introduces a sleep in milliseconds on strategic points, thereby letting the OS catch up.

FFMPeg configuration

Unlike ImageMagick, FFMPeg does not come in different variants and therefore this is a static build provided by the Digibatch installation. Furthermore, this application has very limited configuration parameters, but controls the configuration on the command with which the video is transcoded.

The commandlines for FFMPeg is build dynamically in Digizuite, but they are all located as .xml files on the following location

\\Webserver\Storage\DMM\Profiles\Video

Such a .xml may look as follows

All the %% tokens are replaced with values defined on the resulting format definition, however, there are a few commands that may help limit FFMPeg

The most important parameter is the following

-threads <number of threads>. This parameter controls the amount of threads used by FFMpeg. The default is 1.5 * Cores of the machine.

There are a lot of possible options for FFMPeg and for a full list of the parameters we refer to FFMPegs documentation