{"id":75,"date":"2016-06-07T23:40:52","date_gmt":"2016-06-08T03:40:52","guid":{"rendered":"http:\/\/williamkennerly.com\/blog\/?p=75"},"modified":"2016-06-07T23:40:52","modified_gmt":"2016-06-08T03:40:52","slug":"writing-scripts-to-submit-jobs-on-comet-using-the-slurm-queue-manager","status":"publish","type":"post","link":"http:\/\/williamkennerly.com\/blog\/writing-scripts-to-submit-jobs-on-comet-using-the-slurm-queue-manager\/","title":{"rendered":"Writing scripts to submit jobs on comet using the slurm queue manager"},"content":{"rendered":"

Comet is a huge cluster of thousands of computing nodes, and the queue manager software called “slurm” is what handles all the requests, directs each job to a specific node(s), and then lets you know when its done. In a prior post I showed the basic slurm commands to submit a job and check the queue<\/a>.<\/p>\n

You also need to write a special linux bash script that contains a bunch of slurm configurations, and also the linux commands to actually run your calculation. \u00a0This is easiest show by example, and I’ll show two: \u00a0one for a Gaussian job, and another for an AMBER job.<\/p>\n


\n#!\/bin\/bash
\n#SBATCH -t 10:00:00
\n#SBATCH --job-name=\"gaussian\"
\n#SBATCH --output=\"gaussian.%j.%N.out\"
\n#SBATCH --partition=batch
\n#SBATCH --nodes=1
\n#SBATCH --ntasks-per-node=1
\n#SBATCH --export=ALL<\/code><\/p>\n


\nnprocshared=1
\njobfile=YOURINPUTFILE \u00a0#assumes you use a gjf extension (which is added below)
\njobfile=$jobfile.gjf
\noutputfile=$jobfile.out<\/code><\/p>\n

export GAUSS_SCRDIR=\/scratch\/$USER\/$SLURM_JOBID
\n. \/etc\/profile.d\/modules.sh
\nmodule load gaussian
\nexe=`which g09`
\nexport OMP_NUM_THREADS=$nprocshared
\n\/usr\/bin\/time $exe < $jobfile > $outputfile<\/code><\/p><\/blockquote>\n

If you copy this file exactly and then modify just the important parts, you can submit your own Gaussian jobs quickly. \u00a0 But its worth knowing what these commands do. \u00a0All the SBATCH commands are slurm settings. \u00a0Most important is the line with the “t” flag which sets the wall time you think the job will require. \u00a0(“Wall time” means the same thing as “real time”, as if you were watching the clock on your wall.) \u00a0If your job winds up going longer than your set wall time, then slurm will automatically cancel your job even if it didn’t finish—so don’t underestimate. \u00a0 The other important slurm flags are for “nodes” and “n-tasks-per-node”, which set how many nodes you want the job to use, and how many processors (cores) per node you want your job to use, respectively. \u00a0For Gaussian jobs, you always want to use nodes=1. \u00a0You can start with tasks-per-node=1, and then try and increase it up to 12 (the max for comet) to see if your calculation can take advantage of any of Gaussian’s parallel processing algorithms. \u00a0(You would also need to add a %nprocshared=8<\/code> line to the .gjf file. \u00a0It doesn’t always work that well, meaning your simply wasting processing time that we have to pay for.)<\/p>\n

The commands at the bottom are bash linux commands that set up some important variables, including the stem of the filename for your Gaussian job file. \u00a0Eventually the script then loads the Gaussian module, and with the last line, submits your job.<\/p>\n

If you’re using AMBER for molecular dynamics simulations, here’s a simple slurm script you can copy:<\/p>\n

#!\/bin\/bash -l
\n#SBATCH -t 01:10:00
\n#SBATCH --job-name=\"amber\"
\n#SBATCH --output=\"oamber.%j\"
\n#SBATCH --error=\"eamber.%j\"
\n#SBATCH --partition=\"compute\"
\n#SBATCH --nnodes=1
\n#SBATCH --ntasks-per-node=1
\n#SBATCH --no-requeue
\n#SBATCH --mail-type=END
\n#SBATCH --mail-user=username@skidmore.edu<\/code><\/p>\n

module load amber<\/code><\/p>\n

ibrun sander.MPI -np 1 -O -i equil4.in -p 2mlt.top -c 2mlt_equil3.crd -r 2mlt_equil4.crd -ref 2mlt_equil3.crd -x 2mlt_equil4.trj -o mdout<\/code><\/p><\/blockquote>\n

The SBATCH commands do the same job of configuring slurm, though the “mail-type” and “mail-user” options show how you can have slurm email you when a job finishes. In this example, the last ibrun<\/code> command is the one that actually runs AMBER (a sander job) and sets up all of its settings, input files, output files, and other necessities. Those must be changed for each new job.<\/p>\n

Here is some specific information and examples for comet at the San Diego Supercomputer Center<\/a>, as well as a complete list of all the slurm options you can use<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

Comet is a huge cluster of thousands of computing nodes, and the queue manager software called “slurm” is what handles all the requests, directs each job to a specific node(s), and then lets you know when its done. In a … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[3,2],"class_list":["post-75","post","type-post","status-publish","format-standard","hentry","category-computing","tag-comet","tag-slurm"],"_links":{"self":[{"href":"http:\/\/williamkennerly.com\/blog\/wp-json\/wp\/v2\/posts\/75","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/williamkennerly.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/williamkennerly.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/williamkennerly.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/williamkennerly.com\/blog\/wp-json\/wp\/v2\/comments?post=75"}],"version-history":[{"count":6,"href":"http:\/\/williamkennerly.com\/blog\/wp-json\/wp\/v2\/posts\/75\/revisions"}],"predecessor-version":[{"id":88,"href":"http:\/\/williamkennerly.com\/blog\/wp-json\/wp\/v2\/posts\/75\/revisions\/88"}],"wp:attachment":[{"href":"http:\/\/williamkennerly.com\/blog\/wp-json\/wp\/v2\/media?parent=75"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/williamkennerly.com\/blog\/wp-json\/wp\/v2\/categories?post=75"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/williamkennerly.com\/blog\/wp-json\/wp\/v2\/tags?post=75"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}