Run Matlab Code in Batch Mode on GPU of HPC - HPC at UCI / OIT

<strong>Run</strong> <strong>Matlab</strong> <strong>Code</strong> <strong>in</strong> <strong>Batch</strong> <strong>Mode</strong> on GPU of HPCShiwei LanAugust 30, 2012This short note illustrates technical issues runn<strong>in</strong>g <strong>Matlab</strong> code <strong>in</strong> batch mode onGPU of HPC. It only summaries personal experience on Mac OS X and the best helperis always google.1 <strong>Run</strong> <strong>Matlab</strong> <strong>in</strong> Interactive <strong>Mode</strong>It is sometimes necessary to run <strong>Matlab</strong> <strong>in</strong> <strong>in</strong>teractive mode for test<strong>in</strong>g and debugg<strong>in</strong>gpurposes. This is especially true when your <strong>Matlab</strong> code handles large data set andrequires higher comput<strong>in</strong>g power like GPU. To do so, you have to log on GPU node <strong>in</strong><strong>in</strong>teractive way via x11 (x-quartz for Mounta<strong>in</strong> Lion, but you have to <strong>in</strong>stall it):ssh -Y youraccount@hpc.oit.uci.eduAfter <strong>in</strong>putt<strong>in</strong>g password you are at log-<strong>in</strong> node, then go to GPU node byqrsh -q gpuThen you have to load <strong>Matlab</strong>, available version 2011b:module load MATLAB/r2011bAnd then enter <strong>Matlab</strong> <strong>in</strong> <strong>in</strong>teractive way:matlabAfter the above command, a <strong>Matlab</strong> w<strong>in</strong>dow will pop-up and you can debug any<strong>Matlab</strong> code as you do locally.1

Note on runn<strong>in</strong>g <strong>Matlab</strong> code on GPU2 <strong>Run</strong> <strong>Matlab</strong> <strong>Code</strong> <strong>in</strong> <strong>Batch</strong> <strong>Mode</strong>It is cool to run <strong>Matlab</strong> code <strong>in</strong> batch mode so that you don’t have to wait <strong>in</strong> <strong>in</strong>teractivemode without clos<strong>in</strong>g your computer. However, you CANNOT directly wrap yourcommand ’matlab -r yourcode.m’ <strong>in</strong> bash file because <strong>Matlab</strong> won’t let it run. You haveto compile it first.In the follow<strong>in</strong>g I quote an illustration from BDUC(17.9) to show how to compile<strong>Matlab</strong> code us<strong>in</strong>g <strong>Matlab</strong> command mcc:3 <strong>Run</strong> <strong>Matlab</strong> <strong>Code</strong> on GPUUnlike CUDA, <strong>Matlab</strong> provides easy <strong>in</strong>terface to distribute data to GPU by simplyclaim<strong>in</strong>g your data as ’gpuArray’. Any supported rout<strong>in</strong>e <strong>in</strong>volv<strong>in</strong>g gpuArray data (notall participat<strong>in</strong>g data are required to be gpuArray) will be automatically calculatedAugust 30, 2012 2

Note on runn<strong>in</strong>g <strong>Matlab</strong> code on GPUon GPU. By now, <strong>Matlab</strong> 2012a has a list of rout<strong>in</strong>es like ’<strong>in</strong>v’, ’chol’, ’fft’ etc. support<strong>in</strong>ggpuArray. For those not yet support<strong>in</strong>g gpuArray, you can always first collectby command ’gather’ the gpuArray as regular type data and then do the calculationwith regular data. A start<strong>in</strong>g helper would be the official <strong>Matlab</strong> page about GPU:http://www.mathworks.com/help/toolbox/distcomp/bsic4fr-1.html, where you can alsof<strong>in</strong>d the list of rout<strong>in</strong>es support<strong>in</strong>g gpuArray.Another important <strong>Matlab</strong> command regard<strong>in</strong>g GPU is ’gpuDevice’, with whichyou can get <strong>in</strong>formation of the GPU cards such as ’name’, ’<strong>in</strong>dex’, ’FreeMemory’ etc.Currently there are 4 GPU cards on the GPU node of HPC, by default the <strong>Matlab</strong> willchoose the first available GPU card, however this can be changed by specify<strong>in</strong>g GPUcard <strong>in</strong>dex i: ’gpuDevice(i)’, especially helpful after you <strong>in</strong>quire the memory left from’FreeMemory’. Refer to the official document on gpuDevice.Below is a sample <strong>Matlab</strong> code run on GPU:function[]=<strong>Matlab</strong><strong>Code</strong>()gpuDevice(2) % choose the second GPU cardData = load(’XX.mat’); % Data records two long vectors x, y of size 1e4x=gpuArray(Data.x); y=Data.y; % x is on GPU card but y is notz=x.*y; % This will be done on GPU card and z is also stored as gpuArrayw=y’*repmat(gather(z),1,10); % S<strong>in</strong>ce repeat is not yet support<strong>in</strong>g gpuArray on matlab 2011b,we have to gather z from GPU so it can be applied to repmat and calculated as regular data.end4 <strong>Run</strong> <strong>Matlab</strong> <strong>Code</strong> <strong>in</strong> <strong>Batch</strong> <strong>Mode</strong> on GPUGPU node is now under Grid Eng<strong>in</strong>e, the same as BDUC, which is good for thosefamiliar with BDUC or GE. Remember ’module load GE’ if you have not set it default<strong>in</strong> your ’.bashrc’. Now we are ready to run compiled <strong>Matlab</strong> code <strong>in</strong> batch mode onGPU by submitt<strong>in</strong>g your job to queue ’gpu’.A bash script do<strong>in</strong>g all th<strong>in</strong>g (compil<strong>in</strong>g <strong>Matlab</strong> code, specify<strong>in</strong>g request of batchjob) together is provided as follows:August 30, 2012 3

Note on runn<strong>in</strong>g <strong>Matlab</strong> code on GPU#!/b<strong>in</strong>/bash#$ -N matlabjobname # this name shows <strong>in</strong> qstat#$ -q gpu # submit job to queue ’gpu’#$ -l gpu=1 # request 1 GPU card#$ -pe gpu-node-cores 6 # run with parallel environment ’gpu-node-core’ request<strong>in</strong>g 6 nodecorescd yourwork<strong>in</strong>gdirectory # specify your work<strong>in</strong>g directorymodule load MATLAB/r2011b # load the MATLAB environmentmcc -m matlabcode.m -I directorytoothernecessarydocs; # compile your <strong>Matlab</strong> code./run matlabcode.sh /data/apps/MATLAB/r2011b # run the generated bash to let run your<strong>Matlab</strong> code <strong>in</strong> batch modeNow save your script as ’matlab-batch.sh’ <strong>in</strong> the same folder of your <strong>Matlab</strong> code,and submit it <strong>in</strong> your x-11 or other term<strong>in</strong>al:qsub matlab-batch.shAnd you can always check the status of your job byqstat -u useridError and out put files with same name ’matlab-batch.e (or o)’ will also be generated<strong>in</strong> the same folder for your reference.Hooray! Enjoy!August 30, 2012 4