Programming with DRMAA - Open Grid Forum

OGF25/EGEE User Forum
Programming with DRMAA
DRMAA + WS + JSDL
Krzyszof Kurowski
Paweł Lichocki
Mariusz Mamoński
{krzysztof.kurowski,lichocki,mamonski}@man.poznan.pl

Agenda
•introduction and motivations
•DRMAA
•idea
•overview
•routines
•Academic point of view
•running numerical algorithms
•boosting performance
•exemplary application
•Industrial point of view
•developing middleware
•enhanced security, reliability and functionality
•real-life applications

Introduction … back in Gridlab times
•What we liked:
• people, parties and results ;-)
• idea = GAT + middleware
• new scenarios and user driven
use cases, e.g. Zakopane/migration
• a feedback we provided for O(G)GF
•What we did not like:
• the way middleware was integrated with DRMS
• script-based job managers, new versions/releases, ...
• security model, no support for advanced AAA scenarios
• poor quality, no docs, many bugs and problems with
deployment and support (at that time ;-)
• we had to wait for OGSA, WSRF, … and we liked C and WS
3

Motivations
•GridLab feedback:
•GAT -> a new OGF standard: SAGA
•Middleware -> OGF DRMAA, JSDL, HPC-Profile, ...
•What was really important for our middleware:
• performance (e.g. one million job submissions per day)
stability and portability (e.g. ANSI C not Java)
• simple well accepted standards not only
recommendations or best practices
• flexibility, new security models, …
• support for local and external users performing relatively
simple job submission and monitoring operations (more
advanced scenarios we were discussing under a new
working group in OGF), thus DRMAA
4

DRMAA idea
Application
(binary file)
Application
(binary file)
specific
script
specific
script
specific
script
DRMAA
application
unified API
specific
commands
specific
commands
specific
commands
vendor
DRMAA lib
vendor
DRMAA lib
vendor
DRMAA lib
DRMS 1 DRMS 2 DRMS 3
DRMS 1 DRMS 2 DRMS 3

Overview
•technically DRMAA may be seen as a substitute for the
dedicated scripts for job handling
•from a broader perspective DRMAA is a type of parallel
programming model (like OpenMP or MPI)
•benefits
•provides compatibility with all major DRMSs
•eases the implementation effort and deployment
•good at use cases such as workflows or parameter sweep
•pitfalls
•DRMAA application must be run from the submit host of
the chosen DRMS
•DRMAA application uses only one DRMS at a time

Routines in a nutshell
•session handling
•drmaa_init
•drmaa_exit
•job submission
•job_template routines
•drmaa_submit_job
•drmaa_submit_bulk_jobs
•job monitoring and control
•drmaa_job_ps
•drmaa_control
•job synchronization
•drmaa_wait
•drmaa_synchronize
init
set template
submit job (template)
wait (job)
exit

DRMAA academic use case
•Imagine an MPSD algorithm
•It does not saturate the provided computational power
•There exist different versions, each of them suitable in
other specific case
•It is a crucial part of a very time-demanding application
•It is (practically) impossible to know a priori which
version to use (in order to minimize
the execution time)
•Such algorithms exist, for example
•Simplex - a popular algorithm for
numerical solution of the linear
programming problem
•Hyper-heuristics - an approach of choosing appropriate
heuristic basing on instance characteristic

Example
•Observations
•many methods - standard, revised, Bartels-Golub, sparse
Bartels-Golub, Reid’s Method, ...
•each method suits best different type of data (e.g. sparse
or dense matrices)
•the differences in execution time may be huge
•Problem
•one should analyze the input and choose the method
which seems to be most suitable, but this takes time and
might be incorrect since there are no clear criterias
•Solution
•run all methods, wait for the first to finish, gather
results, terminate other methods

Application flow and assumpptions
•the scheme presents the idea of
the DRMAA application
•in the source code we assume
•the binary file name is simplexN
•the binary takes as an argument
the path to the input file
•the input file is named data.in
•the input file is accessible on all
execution hosts in the working
directory (otherwise a file-staging
must be used)
•we ignore potential errors and
failures of DRMAA calls
init
for i = [0..N)
set i-th template
submit job (i-th template)
j = wait (ANY)
for i = [0..N) and i != j
terminate job (i-th)
exit

Application source code 1/2
int i;
char e[DRMAA_ERROR_STRING_BUFFER];
size_t s = DRMAA_ERROR_STRING_BUFFER;
drmaa_init( NULL, e, s );
drmaa_job_template_t *jt = NULL;
drmaa_allocate_job_template( &jt, e, s );
const char *args[2] = { "data.in", NULL };
drmaa_set_vector_attribute( jt, DRMAA_V_ARGV, args, e, s );
init
allocate template
set generic template
char jobid[ N ][DRMAA_JOBNAME_BUFFER];
for (i = 0; i < N; ++i) {
for i = [0..N)
char cmd[] = "simplex_";
cmd[6] = i + 48;
set i-th template
drmaa_set_attribute( jt, DRMAA_REMOTE_COMMAND, cmd, e, s );
drmaa_run_job( jobid[ i ], DRMAA_JOBNAME_BUFFER, jt, e, s );
submit job (i-th template)
}
drmaa_delete_job_template( jt, e, s );
delete template

Application source code 2/2
for (i = 0; i < N; ++i) {
char jobid_out[DRMAA_JOBNAME_BUFFER];
int status = 0, aborted = 0, exited = 0;
drmaa_attr_values_t *rusage = NULL;
drmaa_wait( DRMAA_JOB_IDS_SESSION_ANY, jobid_out,
DRMAA_JOBNAME_BUFFER, &status,
DRMAA_TIMEOUT_WAIT_FOREVER, &rusage, e, s );
drmaa_wifaborted( &aborted, status, NULL, 0 );
if (aborted != 1) {
drmaa_wifexited( &exited, status, NULL, 0 );
if (exited == 1)
break;
}
}
for (i = 0; i < N; ++i)
drmaa_control( jobid[ i ], DRMAA_CONTROL_TERMINATE, e, s );
for i = [0..N)
j = wait (ANY)
check if job exited normally
if yes do not wait again
for i = [0..N) /*and i != j*/
terminate job (i-th)
drmaa_exit(e, s);
exit

Advanced programming in DRMAA
•The solution was to use drmaa_wait with
DRMAA_JOB_IDS_SESSION_ANY. However waiting for any
“normally terminated” job is not that straightforward
•Other limitations
•The previous approach allows us to run many algorithms
simultaneously, but uses only one DRMS
•The DRMAA application must be run from the submission
host of the chosen DRMS
•DRMAA does not cover issues regarding resource requests
•Questions
•What if we have access to many separated clusters?
•What about security?
•Solution
•map DRMAA functionalities to web-services along with JSDL

Industrial approach - SMOA Computing
• Successor of the OpenDSP (Open DRMAA Service Provider)
• Web Service interface to DRMAA compliant systems
• Adds authentication, authorization and accounting layers
which are out of scope DRMAA specification
• Robust implementation (C, gSOAP toolkit)
• Modular architecture - new scenarios can be realized by
addition of new C/Python modules)
• Use of standard interfaces (JSDL, DRMAA, ODBC, BES HPC
Basic Profile) - easier integration and maintenance
• https://sourceforge.net/projects/smoa-project

JSDL to DRMAA mapping
• ➞ DRMAA_JOB_NAME
• ➞ DRMAA_REMOTE_COMMAND
•* ➞ DRMAA_V_ARGV
•* ➞ DRMAA_V_ENV
• ➞ DRMAA_WD
• ➞ DRMAA_INPUT_PATH
• ➞ DRMAA_OUTPUT_PATH
• ➞ DRMAA_ERROR_PATH
• other JSDL elements if needed can be mapped to
DRMAA_NATIVE_SPECIFICATION by dedicated SMOA
Computing JSDL Filter module

BES to DRMAA mapping (interfaces)
•CreateActivity ➞ drmaa_run_job
•GetActivitiesStatuses ➞ drmaa_job_ps, drmaa_wait
•TerminateActivities ➞ drmaa_control

Example SMOA realization
My organization
2.
6.
3.
1.
Cluster A
SMOA Computing
DRMAA
Compliant
System
Cluster B
1 - Subscribe
2, 3 - Create Activity
4, 5 - Notify
6 - Terminate Activity
5.
SMOA Computing
4.
SMOA Notification
DRMAA
Compliant
System

OpenDSP use case
G-Render - Grid-based Image
Processing System
•used in TeleHVEM
•virtual laboratory for High
Voltage Electron Microscope
•e-science
•Server system
•Sun Grid Engine (SGE) for
computational grid
•OpenDSP for DRMAA web
service
•Client
•Various image processing
features
sge_execd
GRender
GUI
GRender
GRender
Agent
sge_schedd
sge_qmaster
OpenDSP
gSOAP
stubs

Summary & Future
• The best way to learn how to program in DRMAA is to start
writing your own DRMAA application instead of using native
programming interfaces or scripts for a specific DRMS
• If you have access to many computing resources managed by
different DRMSs you may want to use SDKs to DRMAA Service
Provider (now SMOA Computing) we developed for C/C++,
Java, .NET as well as example tools like Vine/GS toolkit,
Jabber clients, …
• We have already extended DRMAA with a set of generic
advanced reservation APIs (testing now with LSF, SGE, PBSPro)
• We integrated our middleware with well known programming
and execution environments ProActive (Java) and OpenMPI (C/
C++/Python) to manage cluster-to-cluster parallel apps
• It should not be difficult to implement a new SAGA adaptor
for SMOA Computing

Links and literature
•[1] OGF DRMAA Working Group, http://www.drmaa.org
•[2] DRMAA Specification, http://www.ogf.org/documents/GFD.133.pdf
•[3] DRMAA C bindings, https://forge.gridforum.org/sf/docman/do/downloadDocument/
projects.drmaa-wg/docman.root.ggf_13/doc5545
•[4] Grid Engine HOWTOs, http://gridengine.sunsource.net/howto/howto.html#DRMAA
•[5] FedStage DRMAA Wiki, http://wiki.fedstage.com/FedStage%20DRMAA
•[6] FedStage DRMAA for PBS PRO, http://sourceforge.net/projects/pbspro-drmaa
•[7] FedStage DRMAA for LSF, http://sourceforge.net/projects/lsf-drmaa
•[8] JSDL Working Group, http://forge.gridforum.org/sf/sfmain/do/viewProject/
projects.jsdl-wg
•[9] JSDL Specification, http://www.gridforum.org/documents/GFD.56.pdf
•[10] gSOAP project, http://www.cs.fsu.edu/~engelen/soap.html
•[11] FedStage Open DRMAA Service Provider, http://sourceforge.net/projects/opendsp
•[12] TeleHVEM, http://goc.pragma-grid.net/wiki/images/2/2e/Hvem-yeom.pdf
•[13] SMOA Project, http://sf.net/projects/smoa-project

Thank you