Programming with the LEADTOOLS Distributed Computing (Job Processor) SDK

The LEADTOOLS Distributed Computing (Job Processor) SDK provides developers with a complete framework which can be used to create powerful distributed computing applications. By deploying your solution to the Job Processor, you move the resources necessary for your application from the end-user's machine or device to an environment you control. Since you control these resources, your application can scale with user demand, without the need to change the application itself.

After the LEADTOOLS Distributed Computing (Job Processor) SDK is installed on the system, you can begin creating powerful applications which can be deployed to the Job Processor. (Note that LEADTOOLS Distributed Computing features must first be unlocked before its properties, methods, and events can be used. For more information on unlocking LEAD features, refer to Unlocking Special LEAD Features.)

The LEADTOOLS Distributed Computing (Job Processor) SDK can be used in any scenario to perform some type of automated job processing on a farm of computers. This opens up a wide range of opportunities. Consider the following examples:

1. A service where users submit a file to be converted from one format to another. Using a web interface for submitting the file, the actual conversion is performed on a farm of worker machines in the Job Processor. After conversion is complete, the converted file is sent back to the user.

2. A service where users submit images from their desktop or mobile device to be processed for OCR (Optical Character Recognition). After an image is received by the Job Processor service it is processed, OCR is performed, and the results are sent back to the user, to be archived in a database or whatever the application architecture requires.

3. A service to use a farm of worker machines dedicated to crawling websites. The LEADTOOLS Distributed Computing (Job Processor) SDK framework can be used to distribute websites in a queued fashion to the different worker machines, and monitor the crawling process for each website.

LEADTOOLS Distributed Computing (Job Processor) SDK

The LEADTOOLS Distributed Computing (Job Processor) SDK is a complete framework. It can be broken down into 3 individual components, as described below. For your reference, a diagram showing these individual complements and the workflow between them is shown in Figure 1.

Figure 1: A diagram of the LEADTOOLS Distributed Computing (Job Processor) SDK components and the workflow between them.

Clients communicate with the Job Processor via the JobService WCF service, which is hosted on the central server. The JobService allows clients to add, abort, delete, and monitor the status of existing jobs. For more information, refer to the IJobService interface.

Central Server - The central server is the primary interface between the clients and the worker machines, and has the following responsibilities:

1. Host the JobService and WorkerService WCF Services.

2. Store Job Processor settings. This includes which types of jobs each worker will accept, the maximum number of concurrent jobs for each worker, CPU thresholds, etc. For more information, refer to LEADTOOLS Job Processor Settings. Storing Job Processor settings on the central server allows user administrators to make global changes to all worker machines in the Job Processor from a single location.

3. Use the Job Processor database to store all jobs and any information related to each job. The central server may or may not contain the Job Processor database. For simplicity in this discussion, assume the Job Processor database is installed on the central server. For more information, refer to Understanding The LEADTOOLS Job Processor Database.

Worker Machine - The worker machines are the machines that actually perform the work for the Job Processor framework. There is no limit to the number of worker machines in the Job Processor: this depends on the specific needs of your Job Processor. As with the clients, communication between the worker machines and the central server is based on Microsoft WCF services. Each worker machine can be configured to process specific types of jobs. In addition, each machine can have machine-dependent settings for the maximum number of concurrent jobs, maximum CPU threshold, and other related settings. For more information, refer to LEADTOOLS Job Processor Settings.

Each worker machine hosts a windows service, which communicates directly with the central server via the WorkerService WCF Service. The windows service will use the WorkerService to obtain new jobs ready for processing, as well as to update the status of any jobs the worker machine accepts. For added stability, each job is processed in its own separate process spawned by the windows service, which communicates directly with the windows service. The windows service will monitor the process for progress, status updates, etc. In the event the process terminates unexpectedly, hangs, or experiences any other fatal error, the windows service will kill the process if necessary, and provide the developer with the ability to react to these issues via the OnJobTerminated Method, and continue to process new jobs.

When a Windows Service is paused, jobs currently in progress will continue to process, but no new jobs will be accepted until the service is resumed. If the service is stopped, or the machine is shut down, all jobs currently in progress will be terminated, and their status will be rolled back to the last status before the job was started. This allows paused jobs to be attempted again when the service resumes.

Begin creating LEADTOOLS Job Processor clients by adding a web reference to the JobService WCF Service. The JobService WCF Service contains the various interfaces, classes, structures and delegates necessary to create a Job Processor client capable of adding, aborting, deleting, and monitoring the status of existing jobs. For more information, refer to the IJobService interface.

Begin creating LEADTOOLS Job Processor Workers by adding a reference to the Leadtools.JobProcessor.Job.dll assembly. The Leadtools.JobProcessor.Job.dll assembly contains the various interfaces, classes, structures and delegates necessary to create a Job Processor worker capable of processing jobs. For more information, refer to the Creating Job Processor Worker Assemblies.

References

LEADTOOLS Job Processor Settings
Understanding The LEADTOOLS Job Processor Database
Creating Job Processor Worker Assemblies
Deploying LEADTOOLS Distributed Computing (JobProcessor) Applications
Unlocking Special LEAD Features
Job Processor Tutorial: Adding a Job to a Job Processor
Job Processor Tutorial: Creating a Job Processor Worker