Compressor 4 - How the Apple Qmaster System Distributes Batches

background image

How the Apple Qmaster System Distributes Batches

The Apple Qmaster cluster controller determines the most efficient use of the cluster
resources. It makes this determination based on the availability of each service node and
the number of separable parts (described next) of the batch.

Because Apple Qmaster subdivides individual batches across different service nodes, the
work is shared and completed more quickly. And, because this method uses all the service
nodes as much as possible, you avoid under-utilizing your resources.

352

Chapter 29

Apple Qmaster and Distributed Processing

background image

Batches can be distributed to a cluster by the cluster controller in one or both of the
following ways. (Apple Qmaster determines which way is the most efficient for specific
batches, depending on the circumstances.)

The batch is subdivided into data segments: For example, for a render batch, the cluster

controller could divide the frames into groups (segments). Each segment would be
processed in parallel on the service nodes in the cluster.

The batch is subdivided into tasks: For example, for a render batch, the cluster controller

could subdivide the rendering work into different processing tasks. Different tasks
would be run on different service nodes.

Rather than actually moving segments, Apple Qmaster tells the service nodes which
segments to read via the network, where to find them, and what to do with them. Below
is an example of how one batch could be processed in an Apple Qmaster system.

Batch submitted by

Compressor or

Apple Qmaster (job request

for frames 1–30)

Cluster controller

divides and distributes

job to available

service nodes

Service node 1

Service node 2

Service node 3

Instructions specifying

locations of source files

and frames 11–20

Instructions specifying

locations of source files

and frames 21–30

Instructions specifying

locations of source files

and frames 01–10

Processed file (frames 1-30)

placed in specified

destination

In distributing batches, Apple Qmaster uses the technology built in to Mac OS X to locate
services in a cluster on the same IP subnet and to dynamically share and receive
information. Because the computers can continually transmit their current processing
availability status, Apple Qmaster can distribute (load-balance) the workload evenly across
the cluster.

353

Chapter 29

Apple Qmaster and Distributed Processing