Skip to main content

Using the Work Queue Manager

This section discusses %SYSTEM.WorkMgrOpens in a new tab specifically. Also see %SYSTEM.ShardWorkMgrOpens in a new tab in the class reference; both classes share a common API.

To use the Work Queue Manager to perform parallel processing:

  1. Identify the ObjectScript code that you want to process in parallel. See Code Requirements.

  2. Divide your code into units of work.

  3. Create a work queue, which is an instance of the %SYSTEM.WorkMgrOpens in a new tab class. To do so, call the %New() method of the %SYSTEM.WorkMgrOpens in a new tab class. The method returns a work queue.

    You can specify the number of parallel worker jobs to use, or you can use the default, which depends on your machine and operating system. Additionally, if you have created categories, you can specify the category that the jobs should be taken from.

    When you create a work queue, the Work Queue Manager creates the following artifacts:

    • A global that contains information about the work queue such as what namespace the work queue runs in

    • A location and an event queue for the serialized units of work that the work queue must process

    • A location and an event queue for completion events that are created as the work queue finishes processing units of work

  4. Add units of work (also called work items) to the work queue. To do so, you can call the Queue() or QueueCallback() method. As arguments, you pass the name of a class method (or subroutine) and any corresponding arguments.

    Processing begins immediately on items added to the queue.

    If there are more items in the queue than there are worker jobs available to the queue, then the jobs compete to empty the queue. For example, if there are 100 items and four jobs, each job removes an item from the head of the queue, processes it, and then returns to the head of the queue to remove and process another item. This pattern continues until the queue is empty.

    The Work Queue Manager uses the security context of the caller when running a work item.

    When you queue work items, the Work Queue Manager performs the following tasks:

    • Serializes the arguments, security context, and class method or subroutine that comprises the unit of work, and then inserts the serialized data into the global that lists the units of work associated with the work queue.

    • Signals an event on the work queue.

    • If additional worker jobs are required and available to process the units of work, causes a worker job to attach to the work queue and decrements the number of available worker jobs.

  5. Wait for the work to be completed. To do so, you can call the WaitForComplete() method of the work queue.

    The Work Queue Manager then performs the following tasks:

    • Waits for a completion event

    • Displays output such as workload metrics to the terminal

    • Collects any errors related to the unit of work

    • If you added units of work to the work queue using the QueueCallback() method, runs the callback code

  6. Continue processing as appropriate for your application.

Example

The following example shows these basic steps:

    Set queue=##class(%SYSTEM.WorkMgr).%New() 
    For i = 1:1:filelist.Count() {
        Set sc=queue.Queue("..Load",filelist.GetAt(i))
        If $$$ISERR(sc) { 
            Return sc
        }
    }
    Set sc=queue.WaitForComplete()
    If $$$ISERR(sc) { 
        Return sc
    }

The code initializes the Work Queue Manager and then iterates through a list of files. For each file, the code adds a work queue item that loads a file. After adding all the work queue items, the code waits for the work to be completed.

Note:

The %SYSTEM.WorkMgrOpens in a new tab class supports more complex workflows with the methods described in other topics.

Methods for Creating Work Queues

To create work queues, add items, and check for completion, use the following methods of the %SYSTEM.WorkMgrOpens in a new tab class:

%New()
classmethod %New(qspec As %String = "", numberjobs As %Integer, category) as WorkMgr

Creates, initializes, and returns a work queue, which is an instance of the %SYSTEM.WorkMgrOpens in a new tab class that you can use to perform parallel processing. The method accepts the following arguments:

qspec

A string of compiler flags and qualifiers that affect code running within this work queue. See Flags and Qualifiers.

numberjobs

The maximum number of parallel worker jobs to use in this work queue. The default depends on the characteristics of the machine and operating system.

category

The name of the category that supplies the worker jobs to use in this work queue. For more information, see Managing Categories.

The system does not allocate any worker jobs to the queue upon creation. Worker jobs are allocated only after you add a unit of work to the work queue.

Queue()
method Queue(work As %String, args... As %String) as %Status

Adds a unit of work to a work queue. The method accepts the following arguments:

work

The code to execute. In general, the code should return a %StatusOpens in a new tab value to indicate success or failure.

If the code returns a %StatusOpens in a new tab value, you can use the following syntax:

  • ##class(Classname).ClassMethod for a class method, where Classname is the fully qualified name of the class and ClassMethod is the name of the method.

    If the method is in the same class, you can use the syntax ..ClassMethod as shown in the example.

  • $$entry^rtn for a subroutine, where entry is the name of the subroutine and rtn is the name of the routine.

If the code does not return a %StatusOpens in a new tab value, use the following syntax instead:

  • =##class(Classname).ClassMethod for a class method (or =..ClassMethod if the method is in the same class)

  • entry^rtn for a subroutine

See About Units of Work for information about the requirements for units of work.

args

A comma-separated list of arguments for the class method or subroutine. To pass a multidimensional array as an argument, precede that argument with a period as usual so that it is passed by reference.

The size of the data passed in these arguments should be relatively small to make the most of the framework. To pass a large amount of information, use a global instead of an argument.

As you queue units of work, the system allocates worker jobs one at a time up to the numberjobs value that you specified when you created the work queue or up to the default value. Additionally, the security context of the caller is recorded, and each work item runs within that security context.

WaitForComplete()
method WaitForComplete(qspec As %String, errorlog As %String) as %Status

Waits for the work queue to complete all the items and then returns a %StatusOpens in a new tab value to indicate success or failure. The %StatusOpens in a new tab value contains information from all %StatusOpens in a new tab values returned by the work items. The method accepts the following arguments:

qspec

A string of compiler flags and qualifiers. See Compiler Flags and Qualifiers.

errorlog

A string of any error information, which is returned as output.

Properties of Work Queues

Each work queue (or instance of %SYSTEM.WorkMgrOpens in a new tab) has the following properties:

NumWorkers

The number of worker jobs assigned to the work queue.

NumActiveWorkers

The number of currently active workers.

Returning Information

The work units can return information (other than status), which is helpful especially in shard queue manager situations where communication with the parent process may not be simple. To do this, the work units can write to the public %result multidimensional array. The caller can access this array in either of two ways:

  • This array is returned by reference in the WaitOne() method.

  • The variable %result is available within the QueueCallback() method.

FeedbackOpens in a new tab