This document primarily presents a quick start guide to the usage of the Gina Cody
+ This document primarily presents a quick start guide to the usage of the Gina Cody
School of Engineering and Computer Science compute server farm called “Speed” – the
GCS ENCS Speed cluster, managed by HPC/NAG of GCS ENCS, Concordia University,
Montreal, Canada.
@@ -39,76 +39,78 @@
This document contains basic information required to use “Speed” as well as tips and tricks, +
This document contains basic information required to use “Speed” as well as tips and tricks, examples, and references to projects and papers that have used Speed. User contributions of sample jobs and/or references are welcome. Details are sent to the hpc-ml mailing list. -
+
+
We receive support from the rest of AITS teams, such as NAG, SAG, FIS, and DOG. -
+
We receive support from the rest of AITS teams, such as NAG, SAG, FIS, and DOG. +
+
Prepare them for big clusters:
+Prepare them for big clusters:
+
We have a great number of open-source software available and installed on Speed – various Python, +
We have a great number of open-source software available and installed on Speed – various Python, CUDA versions, C++/Java compilers, OpenGL, OpenFOAM, OpenCV, TensorFlow, OpenMPI, -OpenISS, MARF [18], etc. There are also a number of commercial packages, subject to -licensing contributions, available, such as MATLAB [7, 17], Abaqus [1], Ansys, Fluent [2], +OpenISS, MARF [21], etc. There are also a number of commercial packages, subject to +licensing contributions, available, such as MATLAB [10, 20], Abaqus [1], Ansys, Fluent [2], etc. -
To see the packages available, run ls -al /encs/pkg/ on speed.encs. -
In particular, there are over 2200 programs available in /encs/bin and /encs/pkg under Scientific +
To see the packages available, run ls -al /encs/pkg/ on speed.encs. +
In particular, there are over 2200 programs available in /encs/bin and /encs/pkg under Scientific Linux 7 (EL7).
Popular concrete examples:
+Popular concrete examples:
Popular examples mentioned (loaded with, module):
+Popular examples mentioned (loaded with, module):
+
After reviewing the “What Speed is” (Section 1.4) and “What Speed is Not” (Section 1.5), request +
After reviewing the “What Speed is” (Section 1.4) and “What Speed is Not” (Section 1.5), request access to the “Speed” cluster by emailing: rt-ex-hpc AT encs.concordia.ca. Faculty and staff may request the access directly. Students must include the following in their message: @@ -263,22 +265,22 @@
+
In these instructions, anything bracketed like so, <>, indicates a label/value to be replaced (the entire +
In these instructions, anything bracketed like so, <>, indicates a label/value to be replaced (the entire bracketed term needs replacement). -
+
Before getting started, please review the “What Speed is” (Section 1.4) and “What Speed is Not” +
Before getting started, please review the “What Speed is” (Section 1.4) and “What Speed is Not” (Section 1.5). Once your GCS ENCS account has been granted access to “Speed”, use your GCS ENCS account credentials to create an SSH connection to speed (an alias for speed-submit.encs.concordia.ca). -
+
Requirements to create connections to Speed: +
Requirements to create connections to Speed: @@ -289,7 +291,7 @@
Open up a terminal window and type in the following SSH command being sure to replace +
Open up a terminal window and type in the following SSH command being sure to replace <ENCSusername> with your ENCS account’s username. @@ -298,18 +300,18 @@
ssh <ENCSusername>@speed.encs.concordia.ca-
-
All users are expected to have a basic understanding of Linux and its commonly used +
+
All users are expected to have a basic understanding of Linux and its commonly used commands. -
+
After creating an SSH connection to “Speed”, you will need to source the “Altair Grid Engine +
After creating an SSH connection to “Speed”, you will need to source the “Altair Grid Engine (AGE)” scheduler’s settings file. Sourcing the settings file will set the environment variables required to execute scheduler commands. -
Based on the UNIX shell type, choose one of the following commands to source the settings +
Based on the UNIX shell type, choose one of the following commands to source the settings file. -
csh/tcsh: +
csh/tcsh: @@ -317,8 +319,8 @@
source /local/pkg/uge-8.6.3/root/default/common/settings.csh-
-
Bourne shell/bash: +
+
Bourne shell/bash: @@ -326,8 +328,8 @@
. /local/pkg/uge-8.6.3/root/default/common/settings.sh-
-
In order to set up the default ENCS bash shell, executing the following command is also +
+
In order to set up the default ENCS bash shell, executing the following command is also required: @@ -336,10 +338,10 @@
printenv ORGANIZATION | grep -qw ENCS || . /encs/Share/bash/profile-
-
To verify that you have access to the scheduler commands execute qstat -f -u "*". If an error is +
+
To verify that you have access to the scheduler commands execute qstat -f -u "*". If an error is returned, attempt sourcing the settings file again. -
The next step is to copy a job template to your home directory and to set up your cluster-specific +
The next step is to copy a job template to your home directory and to set up your cluster-specific storage. Execute the following command from within your home directory. (To move to your home directory, type cd at the Linux prompt and press Enter.) @@ -349,15 +351,15 @@
cp /home/n/nul-uge/template.sh . && mkdir /speed-scratch/$USER-
-
Tip: Add the source command to your shell-startup script. -
Tip: the default shell for GCS ENCS users is tcsh. If you would like to use bash, please contact +
+
Tip: Add the source command to your shell-startup script. +
Tip: the default shell for GCS ENCS users is tcsh. If you would like to use bash, please contact rt-ex-hpc AT encs.concordia.ca. -
For new ENCS Users, and/or those who don’t have a shell-startup script, based on your shell +
For new ENCS Users, and/or those who don’t have a shell-startup script, based on your shell type use one of the following commands to copy a start up script from nul-uge’s. home directory to your home directory. (To move to your home directory, type cd at the Linux prompt and press Enter.) -
csh/tcsh: +
csh/tcsh: @@ -365,8 +367,8 @@
cp /home/n/nul-uge/.tcshrc .-
-
Bourne shell/bash: +
+
Bourne shell/bash: @@ -374,11 +376,11 @@
cp /home/n/nul-uge/.bashrc .-
-
Users who already have a shell-startup script, use a text editor, such as vim or emacs, to add the +
+
Users who already have a shell-startup script, use a text editor, such as vim or emacs, to add the source request to your existing shell-startup environment (i.e., to the .tcshrc file in your home directory). -
csh/tcsh: Sample .tcshrc file: +
csh/tcsh: Sample .tcshrc file: @@ -389,8 +391,8 @@
-
Bourne shell/bash: Sample .bashrc file: +
+
Bourne shell/bash: Sample .bashrc file: @@ -402,34 +404,34 @@
-
Note that you will need to either log out and back in, or execute a new shell, for the environment +
+
Note that you will need to either log out and back in, or execute a new shell, for the environment changes in the updated .tcshrc or .bashrc file to be applied (important). -
+
Preparing your job for submission is fairly straightforward. Editing a copy of the template.sh you +
Preparing your job for submission is fairly straightforward. Editing a copy of the template.sh you moved into your home directory during Section 2.1.2 is a good place to start. You can also use a job script example from our GitHub’s (https://github.com/NAG-DevOps/speed-hpc) “src” directory and base your job on it. -
Job scripts are broken into four main sections:
+Job scripts are broken into four main sections:
+
Directives are comments included at the beginning of a job script that set the shell and the options for +
Directives are comments included at the beginning of a job script that set the shell and the options for the job scheduler. -
The shebang directive is always the first line of a script. In your job script, this directive sets +
The shebang directive is always the first line of a script. In your job script, this directive sets which shell your script’s commands will run in. On “Speed”, we recommend that your script use a shell from the /encs/bin directory. -
To use the tcsh shell, start your script with: #!/encs/bin/tcsh -
For bash, start with: #!/encs/bin/bash -
Directives that start with "#$", set the options for the cluster’s “Altair Grid Engine (AGE)” +
To use the tcsh shell, start your script with: #!/encs/bin/tcsh +
For bash, start with: #!/encs/bin/bash +
Directives that start with "#$", set the options for the cluster’s “Altair Grid Engine (AGE)” scheduler. The script template, template.sh, provides the essentials: @@ -442,15 +444,15 @@
-
Replace, <jobname>, with the name that you want your cluster job to have; -cwd, makes the +
+
Replace, <jobname>, with the name that you want your cluster job to have; -cwd, makes the current working directory the “job working directory”, and your standard output file will appear here; -m bea, provides e-mail notifications (begin/end/abort); replace, <corecount>, with the degree of (multithreaded) parallelism (i.e., cores) you attach to your job (up to 32), be sure to delete or comment out the #$ -pe smp parameter if it is not relevant; replace, <memory>, with the value (in GB), that you want your job’s memory space to be (up to 500), and all jobs MUST have a memory-space assignment. -
If you are unsure about memory footprints, err on assigning a generous memory space to your job +
If you are unsure about memory footprints, err on assigning a generous memory space to your job so that it does not get prematurely terminated (the value given to h_vmem is a hard memory ceiling). You can refine h_vmem values for future jobs by monitoring the size of a job’s active memory space on speed-submit with: @@ -461,17 +463,17 @@
-
Memory-footprint values are also provided for completed jobs in the final e-mail notification (as, +
+
Memory-footprint values are also provided for completed jobs in the final e-mail notification (as, “Max vmem”). -
Jobs that request a low-memory footprint are more likely to load on a busy cluster. -
+
Jobs that request a low-memory footprint are more likely to load on a busy cluster. +
As your job will run on a compute or GPU “Speed” node, and not the submit node, any software that +
As your job will run on a compute or GPU “Speed” node, and not the submit node, any software that is needed must be loaded by the job script. Software is loaded within the script just as it would be from the command line. -
To see a list of which modules are available, execute the following from the command line on +
To see a list of which modules are available, execute the following from the command line on speed-submit. @@ -480,8 +482,8 @@
module avail-
-
To list for a particular program (matlab, for example): +
+
To list for a particular program (matlab, for example): @@ -489,8 +491,8 @@
module -t avail matlab-
-
Which, of course, can be shortened to match all that start with a particular letter: +
+
Which, of course, can be shortened to match all that start with a particular letter: @@ -498,8 +500,8 @@
module -t avail m-
-
Insert the following in your script to load the matlab/R2020a) module: +
+
Insert the following in your script to load the matlab/R2020a) module: @@ -507,9 +509,9 @@
module load matlab/R2020a/default-
-
Use, unload, in place of, load, to remove a module from active use. -
To list loaded modules: +
+
Use, unload, in place of, load, to remove a module from active use. +
To list loaded modules: @@ -517,8 +519,8 @@
module list-
-
To purge all software in your working environment: +
+
To purge all software in your working environment: @@ -526,28 +528,28 @@
module purge-
-
Typically, only the module load command will be used in your script. -
+
+
Typically, only the module load command will be used in your script. +
The last part the job script is the scripting that will be executed by the job. This part of +
The last part the job script is the scripting that will be executed by the job. This part of the job script includes all commands required to set up and execute the task your script has been written to do. Any Linux command can be used at this step. This section can be a simple call to an executable or a complex loop which iterates through a series of commands. -
Every software program has a unique execution framework. It is the responsibility of the script’s +
Every software program has a unique execution framework. It is the responsibility of the script’s author (e.g., you) to know what is required for the software used in your script by reviewing the software’s documentation. Regardless of which software your script calls, your script should be written so that the software knows the location of the input and output files as well as the degree of parallelism. Note that the cluster-specific environment variable, NSLOTS, resolves to the value provided to the scheduler in the -pe smp option. -
Jobs which touch data-input and data-output files more than once, should make use of TMPDIR, a +
Jobs which touch data-input and data-output files more than once, should make use of TMPDIR, a scheduler-provided working space almost 1 TB in size. TMPDIR is created when a job starts, and exists on the local disk of the compute node executing your job. Using TMPDIR results in faster I/O operations than those to and from shared storage (which is provided over NFS). -
An sample job script using TMPDIR is available at /home/n/nul-uge/templateTMPDIR.sh: the job +
An sample job script using TMPDIR is available at /home/n/nul-uge/templateTMPDIR.sh: the job is instructed to change to $TMPDIR, to make the new directory input, to copy data from $SGE_O_WORKDIR/references/ to input/ ($SGE_O_WORKDIR represents the current working directory), to make the new directory results, to execute the program (which takes input from @@ -555,10 +557,10 @@
+
Now, let’s look at a basic job script, tcsh.sh in Figure 1 (you can copy it from our GitHub page or +
Now, let’s look at a basic job script, tcsh.sh in Figure 1 (you can copy it from our GitHub page or from /home/n/nul-uge).
#!/encs/bin/tcsh +#!/encs/bin/tcsh #$ -N qsub-test #$ -cwd @@ -585,7 +587,7 @@2.3
The first line is the shell declaration (also know as a shebang) and sets the shell to tcsh. The lines +
The first line is the shell declaration (also know as a shebang) and sets the shell to tcsh. The lines that begin with #$ are directives for the scheduler.
The script then: +
The script then:
The scheduler command, qsub, is used to submit (non-interactive) jobs. From an ssh session on +
The scheduler command, qsub, is used to submit (non-interactive) jobs. From an ssh session on speed-submit, submit this job with qsub ./tcsh.sh. You will see, "Your job X ("qsub-test") has been submitted". The command, qstat, can be used to look at the status of the cluster: qstat -f -u "*". You will see something like this: @@ -659,25 +661,25 @@
+
-
Remember that you only have 30 seconds before the job is essentially over, so if you do not see a +
Remember that you only have 30 seconds before the job is essentially over, so if you do not see a similar output, either adjust the sleep time in the script, or execute the qstat statement more quickly. The qstat output listed above shows you that your job is running on node speed-05, that it has a job number of 144, that it was started at 16:39:30 on 12/03/2018, and that it is a single-core job (the default). -
Once the job finishes, there will be a new file in the directory that the job was started from, with +
Once the job finishes, there will be a new file in the directory that the job was started from, with the syntax of, "job name".o"job number", so in this example the file is, qsub test.o144. This file represents the standard output (and error, if there is any) of the job in question. If you look at the contents of your newly created file, you will see that it contains the output of the, module list command. Important information is often written to this file. -
Congratulations on your first job! +
Congratulations on your first job!
Here are useful job-management commands: +
Here are useful job-management commands:
+
In addition to the basic qsub options presented earlier, there are a few additional options that are +
In addition to the basic qsub options presented earlier, there are a few additional options that are generally useful:
+
Array jobs are those that start a batch job or a parallel job multiple times. Each iteration of the job +
Array jobs are those that start a batch job or a parallel job multiple times. Each iteration of the job array is called a task and receives a unique job ID. -
To submit an array job, use the t option of the qsub command as follows: +
To submit an array job, use the t option of the qsub command as follows: @@ -744,15 +746,15 @@
qsub -t n[-m[:s]] <batch_script>-
-
-t Option Syntax:
++
-t Option Syntax:
Examples:
+Examples:
Output files for Array Jobs: -
The default and output and error-files are job_name.[o|e]job_id and
job_name.[o|e]job_id.task_id. This means that Speed creates an output and an error-file for each
+
Output files for Array Jobs: +
The default and output and error-files are job_name.[o|e]job_id and
job_name.[o|e]job_id.task_id. This means that Speed creates an output and an error-file for each
task generated by the array-job as well as one for the super-ordinate array-job. To alter this behavior
use the -o and -e option of qsub.
-
For more details about Array Job options, please review the manual pages for qsub by executing +
For more details about Array Job options, please review the manual pages for qsub by executing the following at the command line on speed-submit man qsub. -
+
For jobs that can take advantage of multiple machine cores, up to 32 cores (per job) can be requested +
For jobs that can take advantage of multiple machine cores, up to 32 cores (per job) can be requested in your script with: @@ -779,26 +781,26 @@
-
Do not request more cores than you think will be useful, as larger-core jobs +
+
Do not request more cores than you think will be useful, as larger-core jobs are more difficult to schedule. On the flip side, though, if you are going to be running a program that scales out to the maximum single-machine core count available, please (please) request 32 cores, to avoid node oversubscription (i.e., to avoid overloading the CPUs). -
Core count associated with a job appears under, “states”, in the, qstat -f -u "*", +
Core count associated with a job appears under, “states”, in the, qstat -f -u "*", output. -
+
Job sessions can be interactive, instead of batch (script) based. Such sessions can be useful for testing +
Job sessions can be interactive, instead of batch (script) based. Such sessions can be useful for testing and optimising code and resource requirements prior to batch submission. To request an interactive job session, use, qlogin [options], similarly to a qsub command-line job (e.g., qlogin -N qlogin-test -l h_vmem=1G). Note that the options that are available for qsub are not necessarily available for qlogin, notably, -cwd, and, -v. -
+
The scheduler presents a number of environment variables that can be used in your jobs. Three of the +
The scheduler presents a number of environment variables that can be used in your jobs. Three of the more useful are TMPDIR, SGE_O_WORKDIR, and NSLOTS:
In Figure 2 is a sample script, using all three. +
In Figure 2 is a sample script, using all three.
#!/encs/bin/tcsh #$ -N envs #$ -cwd @@ -843,7 +845,7 @@2.10 SSH Keys For MPI
-Some programs effect their parallel processing via MPI (which is a communication protocol). An +
Some programs effect their parallel processing via MPI (which is a communication protocol). An example of such software is Fluent. MPI needs to have ‘passwordless login’ set up, which means SSH keys. In your NFS-mounted home directory:
@@ -858,15 +860,15 @@2.10
- Set file permissions of authorized_keys to 600; of your NFS-mounted home to 700 (note that you likely will not have to do anything here, as most people will have those permissions by default).
+
The following documentation is specific to the Speed HPC Facility at the Gina Cody School of +
The following documentation is specific to the Speed HPC Facility at the Gina Cody School of Engineering and Computer Science. -
+
To create an anaconda environment in your speed-scratch directory, use the prefix option when +
To create an anaconda environment in your speed-scratch directory, use the prefix option when executing conda create. For example, to create an anaconda environment for ai_user, execute the following at the command line: @@ -876,11 +878,11 @@
-
Note: Without the prefix option, the conda create command creates the environment in +
+
Note: Without the prefix option, the conda create command creates the environment in texttta_user’s home directory by default.
-List Environments. To view your conda environments, type: conda info --envs @@ -892,9 +894,9 @@
+
-
Activate an Environment. Activate the environment speedscratcha_usermyconda as follows @@ -903,7 +905,7 @@
After activating your environment, add pip to your environment by using +
After activating your environment, add pip to your environment by using @@ -911,10 +913,10 @@
This will install pip and pip’s dependencies, including python, into the environment. -
Important Note: pip (and pip3) are used to install modules from the python distribution while +
This will install pip and pip’s dependencies, including python, into the environment. +
Important Note: pip (and pip3) are used to install modules from the python distribution while conda install installs modules from anaconda’s repository. -
+
#!/encs/bin/tcsh #$ -N flu10000 #$ -cwd @@ -946,7 +948,7 @@-The job script in Figure 3 runs Fluent in parallel over 32 cores. Of note, we have requested e-mail +
The job script in Figure 3 runs Fluent in parallel over 32 cores. Of note, we have requested e-mail notifications (-m), are defining the parallel environment for, fluent, with, -sgepe smp (very important), and are setting $TMPDIR as the in-job location for the “moment” rfile.out file (in-job, because the last line of the script copies everything from $TMPDIR to a directory in the user’s @@ -956,7 +958,7 @@
2.13 Example Job: efficientdet
-The following steps describing how to create an efficientdet environment on Speed, were submitted by +
The following steps describing how to create an efficientdet environment on Speed, were submitted by a member of Dr. Amer’s research group.
@@ -985,24 +987,24 @@
-
-
+
+
2.14 Java Jobs
-Jobs that call java have a memory overhead, which needs to be taken into account when assigning a +
Jobs that call java have a memory overhead, which needs to be taken into account when assigning a value to h_vmem. Even the most basic java call, java -Xmx1G -version, will need to have, -l h_vmem=5G, with the 4-GB difference representing the memory overhead. Note that this memory overhead grows proportionally with the value of -Xmx. To give you an idea, when -Xmx has a value of 100G, h_vmem has to be at least 106G; for 200G, at least 211G; for 300G, at least 314G. -
+
2.15 Scheduling On The GPU Nodes
-The primary cluster has two GPU nodes, each with six Tesla (CUDA-compatible) P6 cards: each card +
The primary cluster has two GPU nodes, each with six Tesla (CUDA-compatible) P6 cards: each card has 2048 cores and 16GB of RAM. Though note that the P6 is mainly a single-precision card, so unless you need the GPU double precision, double-precision calculations will be faster on a CPU node. -
Job scripts for the GPU queue differ in that they do not need these statements: +
Job scripts for the GPU queue differ in that they do not need these statements: @@ -1011,8 +1013,8 @@
-
But do need this statement, which attaches either a single GPU, or, two GPUs, to the +
+
But do need this statement, which attaches either a single GPU, or, two GPUs, to the job: @@ -1021,11 +1023,11 @@
#$ -l gpu=[1|2]
-
Single-GPU jobs are granted 5 CPU cores and 80GB of system memory, and dual-GPU jobs are +
+
Single-GPU jobs are granted 5 CPU cores and 80GB of system memory, and dual-GPU jobs are granted 10 CPU cores and 160GB of system memory. A total of four GPUs can be actively attached to any one user at any given time. -
Once that your job script is ready, you can submit it to the GPU queue with: +
Once that your job script is ready, you can submit it to the GPU queue with: @@ -1033,8 +1035,8 @@
-
And you can query nvidia-smi on the node that is running your job with: +
+
And you can query nvidia-smi on the node that is running your job with: @@ -1042,8 +1044,8 @@
-
Status of the GPU queue can be queried with: +
+
Status of the GPU queue can be queried with: @@ -1051,16 +1053,16 @@
-
Very important note regarding TensorFlow and PyTorch: if you are planning to run TensorFlow +
+
Very important note regarding TensorFlow and PyTorch: if you are planning to run TensorFlow
and/or PyTorch multi-GPU jobs, do not use the tf.distribute and/or
torch.nn.DataParallel functions, as they will crash the compute node (100% certainty). This
appears to be the current hardware’s architecture’s defect. The workaround is to either manually
effect GPU parallelisation (TensorFlow has an example on how to do this), or to run on a single
GPU.
-
Important -
Users without permission to use the GPU nodes can submit jobs to the g.q queue but those jobs +
Important +
Users without permission to use the GPU nodes can submit jobs to the g.q queue but those jobs will hang and never run. -
There are two GPUs in both speed-05 and speed-17, and one in speed-19. Their availability is +
There are two GPUs in both speed-05 and speed-17, and one in speed-19. Their availability is seen with, qstat -F g (note the capital): @@ -1084,8 +1086,8 @@
This status demonstrates that all five are available (i.e., have not been requested as resources). To +
+
This status demonstrates that all five are available (i.e., have not been requested as resources). To specifically request a GPU node, add, -l g=[#GPUs], to your qsub (statement/script) or qlogin (statement) request. For example, qsub -l h_vmem=1G -l g=1 ./count.sh. You will see that this job has been assigned to one of the GPU nodes: @@ -1105,13 +1107,13 @@
And that there are no more GPUs available on that node (hc:gpu=0). Note that no more than two +
+
And that there are no more GPUs available on that node (hc:gpu=0). Note that no more than two GPUs can be requested for any one job. -
+
When calling CUDA within job scripts, it is important to create a link to the desired CUDA libraries and +
When calling CUDA within job scripts, it is important to create a link to the desired CUDA libraries and set the runtime link path to the same libraries. For example, to use the cuda-11.5 libraries, specify the following in your Makefile. @@ -1121,16 +1123,16 @@
-L/encs/pkg/cuda-11.5/root/lib64 -Wl,-rpath,/encs/pkg/cuda-11.5/root/lib64-
-
In your job script, specify the version of gcc to use prior to calling cuda. For example: module
+
+ In your job script, specify the version of gcc to use prior to calling cuda. For example: module
load gcc/8.4 or module load gcc/9.3
-
+
It is not possible to create a qlogin session on to a node in the GPU Queue (g.q). As direct logins
+ It is not possible to create a qlogin session on to a node in the GPU Queue (g.q). As direct logins
to these nodes is not available, jobs must be submitted to the GPU Queue in order to compile and
link.
- We have several versions of CUDA installed in:
+ We have several versions of CUDA installed in:
@@ -1140,19 +1142,151 @@ 2.15.2 Special Notes for sending CUDA jobs to the GPU Queue
-
-
For CUDA to compile properly for the GPU queue, edit your Makefile replacing usrlocalcuda +
+
For CUDA to compile properly for the GPU queue, edit your Makefile replacing usrlocalcuda with one of the above. -
+
+
+These represent more comprehensive research-like examples of jobs for computer vision and other +tasks with a lot longer runtime (a subject to the number of epochs and other parameters) derive from +the actual research works of students and their theses. These jobs require the use of CUDA +and GPUs. These examples are available as “native” jobs on Speed and as Singularity +containers. +
+OpenISS and REID + + The example openiss-reid-speed.sh illustrates a job for a computer-vision based person +re-identification (e.g., motion capture-based tracking for stage performance) part of the OpenISS +project by Haotao Lai [7] using TensorFlow and Keras. The fork of the original repo [9] adjusted to +to run on Speed is here:
-The cluster is, “first come, first served”, until it fills, and then job position in the queue is +
+and its detailed description on how to run it on Speed is in the README: +
+ + + + +OpenISS and YOLOv3 + + The related code using YOLOv3 framework is in the the fork of the original repo [8] adjusted to +to run on Speed is here: +
+ +Its example job scripts can run on both CPUs and GPUs, as well as interactively using +TensorFlow: +
+The detailed description on how to run these on Speed is in the README at: +
+ ++
+If the /encs software tree does not have a required software instantaneously available, another option +is to run Singularity containers. We run EL7 flavor of Linux, and if some projects require Ubuntu or +other distributions, there is a possibility to run that software as a container, including the ones +translated from Docker. +
The example lambdal-singularity.sh showcases an immediate use of a container built for the +Ubuntu-based LambdaLabs software stack, originally built as a Docker image then pulled +in as a Singularity container that is immediately available for use as that job example +illustrates. The source material used for the docker image was our fork of their official repo: +https://github.com/NAG-DevOps/lambda-stack-dockerfiles + + + +
NOTE: It is important if you make your own containers or pull from DockerHub, use your +/speed-scratch/$USER directory as these images may easily consume gigs of space in your home +directory and you’d run out of quota there very fast. +
TIP: To check for your quota, and the corresponding commands to find big files, see: +https://www.concordia.ca/ginacody/aits/encs-data-storage.html +
We likewise built equivalent OpenISS (Section 2.15.3) containers from their Docker +counter parts as they were used for teaching and research [11]. The images from +https://github.com/NAG-DevOps/openiss-dockerfiles and their DockerHub equivalents +https://hub.docker.com/u/openiss are found in the same public directory on +/speed-scratch/nag-public as the LambdaLabs Singularity image. They all have .sif extension. +Some of them can be ran in both batch or interactive mode, some make more sense to +run interactively. They cover some basics with CUDA, OpenGL rendering, and computer +vision tasks as examples from the OpenISS library and other libraries, including the base +images that use diffrent distros. We also include Jupyter notebook example with Conda +support. + + + +
++/speed-scratch/nag-public: + +openiss-cuda-conda-jupyter.sif +openiss-cuda-devicequery.sif +openiss-opengl-base.sif +openiss-opengl-cubes.sif +openiss-opengl-triangle.sif +openiss-reid.sif +openiss-xeyes.sif ++
+
The currently recommended version of Singularity is singularity/3.10.4/default. +
This section comprises an introduction to working with Singularity, its containers, and what can +and cannot be done with Singularity on the ENCS infrastructure. It is not intended to be an +exhaustive presentation of Singularity: the program’s authors do a good job of that here: +https://www.sylabs.io/docs/. It also assumes that you have successfully installed Singularity on a +user-managed/personal system (see next paragraph as to why). +
Singularity containers are essentially either built from an existing container, or are built from +scratch. Building from scratch requires a recipe file (think of like a Dockerfile), and the operation must +be effected as root. You will not have root on the ENCS infrastructure, so any built-from-scratch +containers must be created on a user-managed/personal system. Root-level permissions are also +required (in some cases, essential; in others, for proper build functionality) for building from an +existing container. Three types of Singularity containers can be built: file-system; sandbox; squashfs. +The first two are “writable” (meaning that changes can persist after the Singularity session ends). +File-system containers are built around the ext3 file system, and are a read-write “file”, sandbox +containers are essentially a directory in an existing read-write space, and squashfs containers are +a read-only compressed “file”. Note that file-system containers cannot be resized once +built. +
Note that the default build is a squashfs one. Also note what Singularity’s authors have to say +about the builds, “A common workflow is to use the “sandbox” mode for development of +the container, and then build it as a default (squashfs) Singularity image when done.” +File-system containers are considered to be, “legacy”, at this point in time. When built, a +very small overhead is allotted to a file-system container (think, MB), and that cannot be +changed. +
Probably for the most of your workflows you might find there is a Docker container exists for your +tasks, in this case you can use the docker pull function of Singularity as a part of you virtual +environment setup as an interactive job allocation: + + + +
++qlogin +cd /speed-scratch/$USER/ +singularity pull openiss-cuda-devicequery.sif docker://openiss/openiss-cuda-devicequery +INFO: Converting OCI blobs to SIF format +INFO: Starting build... ++
+
This method can be used for converting Docker containers directly on Speed. On GPU nodes make +sure to pass on the --nv flag to Singularity, so its containers could access the GPUs. See the linked +example. +
+
+The cluster is, “first come, first served”, until it fills, and then job position in the queue is based upon past usage. The scheduler does attempt to fill gaps, though, so sometimes a single-core job of lower priority will schedule before a multi-core job of higher priority, for example. -
+
-
Scripts can live in your NFS-provided home, but any substantial data need to be in your +
Scripts can live in your NFS-provided home, but any substantial data need to be in your cluster-specific directory (located at /speed-scratch/<ENCSusername>/). -
NFS is great for acute activity, but is not ideal for chronic activity. Any data that a +
NFS is great for acute activity, but is not ideal for chronic activity. Any data that a job will read more than once should be copied at the start to the scratch disk of a compute node using $TMPDIR (and, perhaps, $SGE_O_WORKDIR), any intermediary job data should be produced in $TMPDIR, and once a job is near to finishing, those data should - be copied to your NFS-mounted home (or other NFS-mounted space) from $TMPDIR (to, - perhaps, $SGE_O_WORKDIR). In other words, IO-intensive operations should be effected - locally whenever possible, saving network activity for the start and end of jobs. + be copied to your NFS-mounted home (or other NFS-mounted space) from $TMPDIR (to, + perhaps, $SGE_O_WORKDIR). In other words, IO-intensive operations should be effected + locally whenever possible, saving network activity for the start and end of jobs.
+
-
+
-
HPC Committee’s initial batch about 6 students (end of 2019):
+HPC Committee’s initial batch about 6 students (end of 2019):
NAG’s MAC spoofer analyzer [12, 11], such as https://github.com/smokhov/atsm/tree/master/examples/flucid +
NAG’s MAC spoofer analyzer [15, 14], such as https://github.com/smokhov/atsm/tree/master/examples/flucid
S4 LAB/GIPSY R&D Group’s:
+S4 LAB/GIPSY R&D Group’s:
-
--
-The work “Haotao Lai. An OpenISS framework specialization for deep learning-based + person re-identification. Master’s thesis, Department of Computer Science and + Software Engineering, Concordia University, Montreal, Canada, August 2019. + https://spectrum.library.concordia.ca/id/eprint/985788/” using TensorFlow and Keras + on OpenISS adjusted to run on Speed based on the repositories:
+and theirs forks by the team. +
++
++
++
-
Phase 3 had 4 vidpro nodes added from Dr. Amer totalling 6x P6 and 6x V100 GPUs +
Phase 3 had 4 vidpro nodes added from Dr. Amer totalling 6x P6 and 6x V100 GPUs added. -
+
-
Phase 2 saw 6x NVIDIA Tesla P6 added and 8x more compute nodes. The P6s replaced 4x of FirePro +
Phase 2 saw 6x NVIDIA Tesla P6 added and 8x more compute nodes. The P6s replaced 4x of FirePro S7150. -
+ + + +
-
Phase 1 of Speed was of the following configuration: +
Phase 1 of Speed was of the following configuration:
+
-
+
-
All Speed users are expected to have a basic understanding of Linux and its commonly used +
All Speed users are expected to have a basic understanding of Linux and its commonly used commands. -
+
-
Software Carpentry provides free resources to learn software, including a workshop on the Unix shell. +
Software Carpentry provides free resources to learn software, including a workshop on the Unix shell. https://software-carpentry.org/lessons/ -
+
-
There are a number of Udemy courses, including free ones, that will assist you in learning Linux. +
There are a number of Udemy courses, including free ones, that will assist you in learning Linux. Active Concordia faculty, staff and students have access to Udemy courses such as Linux Mastery: Master the Linux Command Line in 11.5 Hours is a good starting point for beginners. Visit https://www.concordia.ca/it/services/udemy.html to learn how Concordians may access Udemy. -
+ + + +
-
This section describes how to use the “bash shell” on Speed. Review Section 2.1.2 to ensure that your +
This section describes how to use the “bash shell” on Speed. Review Section 2.1.2 to ensure that your bash environment is set up. -
+
-
In order to set your login shell to bash on Speed, your login shell on all GCS servers must be changed +
In order to set your login shell to bash on Speed, your login shell on all GCS servers must be changed to bash. To make this change, create a ticket with the Service Desk (or email help at concordia.ca) to request that bash become your default login shell for your ENCS user account on all GCS servers. -
+
-
To move to the bash shell, type bash at the command prompt. For example: +
To move to the bash shell, type bash at the command prompt. For example:
--[speed-27] [/home/a/a_user] > bash ++[speed-submit] [/home/a/a_user] > bash bash-4.4$ echo $0 bash--
Note how the command prompt changed from [speed-27] [/home/a/a_user] > to bash-4.4$ -after entering the bash shell. -
+
+
Note how the command prompt changed from [speed-submit] [/home/a/a_user] > to +bash-4.4$ after entering the bash shell. +
-
B.2.3 How do I run scripts written in bash on Speed?
-To execute bash scripts on Speed: +
B.2.3 How do I run scripts written in bash on Speed?
+To execute bash scripts on Speed:
-
-- Ensure that the shebang of your bash job script is #/encs/bin/bash! +
- Ensure that the shebang of your bash job script is #/encs/bin/bash!
-- Use the qsub command to submit your job script to the scheduler.
The Speed GitHub contains a sample bash job script. -
+
The Speed GitHub contains a sample bash job script. +
-
+
-
The ‘‘Disk quota exceeded’’ Error occurs when your application has run out of disk space to write +
The ‘‘Disk quota exceeded’’ Error occurs when your application has run out of disk space to write to. On Speed this error can be returned when:
+
-
+
The use local disk space is generally recommended for IO intensive operations. However, as the +
The use local disk space is generally recommended for IO intensive operations. However, as the size of /tmp on speed nodes is 1GB it can be necessary for scripts to store temporary data elsewhere. Review the documentation for each module called within your script to determine how to set working directories for that application. The basic steps for this solution are: @@ -1391,116 +1544,116 @@
Create a working directory in speed-scratch for output files. For example, this +
Create a working directory in speed-scratch for output files. For example, this command will create a subdirectory called output in your speed-scratch directory:
-+mkdir -m 750 /speed-scratch/$USER/output-+
To create a subdirectory for recovery files: +
To create a subdirectory for recovery files:
-+mkdir -m 750 /speed-scratch/$USER/recovery-+
In the above example, $USER is an environment variable containing your ENCS username. -
+
In the above example, $USER is an environment variable containing your ENCS username. +
-
Create directories for recovery, temporary, and configuration files. For example, to create these - directories for your encs user account: +
Create directories for recovery, temporary, and configuration files. For example, to create these + directories for your GCS ENCS user account:
-+mkdir -m 750 -p /speed-scratch/$USER/comsol/{recovery,tmp,config}-+
Add the following command switches to the COMSOL command to use the directories created +
Add the following command switches to the COMSOL command to use the directories created above:
-
+
-recoverydir /speed-scratch/$USER/comsol/recovery
-tmpdir /speed-scratch/$USER/comsol/tmp
-configuration/speed-scratch/$USER/comsol/config
- In the above example, $USER is an environment variable containing your ENCS username. -
+
+In the above example, $USER is an environment variable containing your ENCS username. +
-
By default when adding a python module the /tmp directory is set as the temporary repository for +
By default when adding a python module the /tmp directory is set as the temporary repository for files downloads. The size of the /tmp directory on speed-submit is too small for pytorch. To add a python module
Create your own tmp directory in your speed-scratch direcrtory +
Create your own tmp directory in your speed-scratch directory
-- mkdir /speed-scratch/$USER/tmp ++ mkdir /speed-scratch/$USER/tmp-+
Use the tmp direcrtory you created +
Use the tmp directory you created
-- setenv TMPDIR /speed-scratch/$USER/tmp ++ setenv TMPDIR /speed-scratch/$USER/tmp-+
In the above example, $USER is an environment variable containing your ENCS username. -
+
In the above example, $USER is an environment variable containing your ENCS username. +
-
When a job with a job id of 1234 is running, the status of that job can be tracked using +
When a job with a job id of 1234 is running, the status of that job can be tracked using ‘qstat -j 1234‘. Likewise, if the job is pending, the ‘qstat -j 1234‘ command will report as to why the job is not scheduled or running. Once the job has finished, or has been killed, the qacct command must be used to query the job’s status, e.g., ‘qaact -j [jobid]‘. -
+
-
+
-
It is possible that a (or a number of) the Speed nodes are disabled. Nodes are disabled if they require +
It is possible that a (or a number of) the Speed nodes are disabled. Nodes are disabled if they require maintenance. To verify if Speed nodes are disabled, request the current list of disabled nodes from qstat.
-+qstat -f -qs d queuename qtype resv/used/tot. load_avg arch states --------------------------------------------------------------------------------- @@ -1518,19 +1671,20 @@-B.5.1 --------------------------------------------------------------------------------- s.q@speed-36.encs.concordia.ca BIP 0/0/32 0.03 lx-amd64 d
-
Note how the all of the Speed nodes in the above list have a state of d, or disabled. -
Your job will run once the maintenance has been completed and the disabled nodes have been +
+
Note how the all of the Speed nodes in the above list have a state of d, or disabled. +
Your job will run once the maintenance has been completed and the disabled nodes have been enabled. +
-
B.5.2 Error in job submit request.
-It is possible that your job is pending, because the job requested resources that are not available +
B.5.2 Error in job submit request.
+It is possible that your job is pending, because the job requested resources that are not available within Speed. To verify why pending job with job id 1234 is not running, execute ‘qstat -j 1234‘ and review the messages in the scheduling info: section. -
+
-
C Sister Facilities
-Below is a list of resources and facilities similar to Speed at various capacities. Depending on your +
C Sister Facilities
+Below is a list of resources and facilities similar to Speed at various capacities. Depending on your research group and needs, they might be available to you. They are not managed by HPC/NAG of AITS, so contact their respective representatives.
@@ -1541,31 +1695,49 @@C -
apini.encs cluster for teaching and MPI programming (see the corresponding course) - -Computer Science and Software Engineering (CSSE) Virya GPU Cluster (2 nodes totalling - 16 V100 NVIDIA GPUs), contact Alexander Aric at gpu-help AT encs to request access - if you are a CSSE member + apini.encs cluster for teaching and MPI programming (see the corresponding course in + CSSE) -Dr. Maria Amer’s VidPro group’s nodes in Speed with additional V100 and P6 GPUs - (use a.q for those nodes). + Computer Science and Software Engineering (CSSE) Virya GPU Cluster. For CSSE + members only. The cluster has 4 nodes with total of 32 NVIDIA GPUs (a mix of V100s + and A100s). To request access send email to virya.help@concordia.ca. -Dr. Amin Hammad’s construction.encs Lambda Labs station + Dr. Maria Amer’s VidPro group’s nodes in Speed (-01, -03, -25, -27) with additional V100 + and P6 GPUs. -Dr. Hassan Rivaz’s impactlab.encs Lambda Labs station + + -There are various Lambda Labs other GPU servers and like computers acquired by individual + researchers; if you are member of their research group, contact them directly. These resources + are not managed by us.
++
- Dr. Amin Hammad’s construction.encs Lambda Labs station +
+- Dr. Hassan Rivaz’s impactlab.encs Lambda Labs station +
+- Dr. Nizar Bouguila’s xailab.encs Lambda Labs station +
+- Dr. Roch Glitho’s femto.encs server +
+- Dr. Maria Amer’s venom.encs Lambda Labs station +
+- Dr. Leon Wang’s guerrera.encs DGX station
Dr. Ivan Contreras’ servers + Dr. Ivan Contreras’ servers (managed by AITS) + + + -If you are a member of School of Health (formerly PERFORM Center), you may have - access to their local PERFORM’s High Performance Computing (HPC) Cluster. Contact - Thomas Beaudry for details and how to obtain access. + If you are a member of School of Health (formerly PERFORM Center), you may have access to + their local PERFORM’s High Performance Computing (HPC) Cluster. Contact Thomas + Beaudry for details and how to obtain access. -Digital Alliance (Compute Canada / Calcul Quebec) +Digital Research Alliance Canada (Compute Canada / Calcul Quebec), -
https://alliancecan.ca/ +References
+References
@@ -1598,66 +1770,83 @@
C https://github.com/goutamyg/SMAT.
- [7] MathWorks. MATLAB. [online], 2000–2012. http://www.mathworks.com/products/matlab/. + [7] Haotao Lai. An OpenISS framework + specialization for deep learning-based person re-identification. Master’s thesis, Department of + Computer Science and Software Engineering, Concordia University, Montreal, Canada, August + 2019. https://spectrum.library.concordia.ca/id/eprint/985788/.
- [8] Serguei Mokhov, Jonathan Llewellyn, Carlos Alarcon Meza, Tariq Daradkeh, and Gillian + [8] Haotao Lai et al. OpenISS keras-yolo3 v0.1.0, June 2021. + https://github.com/OpenISS/openiss-yolov3. +
++ [9] Haotao Lai et al. Openiss person re-identification baseline v0.1.1, June 2021. + https://github.com/OpenISS/openiss-reid-tfk. + + + +
++ [10] MathWorks. MATLAB. [online], 2000–2012. http://www.mathworks.com/products/matlab/. +
++ [11] Serguei Mokhov, Jonathan Llewellyn, Carlos Alarcon Meza, Tariq Daradkeh, and Gillian Roper. The use of containers in OpenGL, ML and HPC for teaching and research support. In ACM SIGGRAPH 2023 Posters, SIGGRAPH ’23, New York, NY, USA, 2023. ACM. https://doi.org/10.1145/3588028.3603676.
- [9] Serguei A. Mokhov. The use of machine learning with signal- and NLP processing + [12] Serguei A. Mokhov. The use of machine learning with signal- and NLP processing of source code to fingerprint, detect, and classify vulnerabilities and weaknesses with MARFCAT. Technical Report NIST SP 500-283, NIST, October 2011. Report: - - - http://www.nist.gov/manuscript-publication-search.cfm?pub_id=909407, online e-print at http://arxiv.org/abs/1010.2511.
- [10] Serguei A. Mokhov. Intensional Cyberforensics. PhD thesis, Department of Computer Science + [13] Serguei A. Mokhov. Intensional Cyberforensics. PhD thesis, Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada, September 2013. Online at http://arxiv.org/abs/1312.0466.
- [11] Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Automating MAC + [14] Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Automating MAC spoofer evidence gathering and encoding for investigations. In Frederic Cuppens et al., editors, Proceedings of The 7th International Symposium on Foundations & Practice of Security (FPS’14), LNCS 8930, pages 168–183. Springer, November 2014. Full paper.
- [12] Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Toward automated + [15] Serguei A. Mokhov, Michael J. Assels, Joey Paquet, and Mourad Debbabi. Toward automated MAC spoofer investigations. In Proceedings of C3S2E’14, pages 179–184. ACM, August 2014. Short paper.
- [13] Serguei A. Mokhov and Scott Bunnell. Speed server farm: + [16] Serguei A. Mokhov and Scott Bunnell. Speed server farm: Gina Cody School of ENCS HPC facility. [online], 2018–2019. https://docs.google.com/presentation/d/1bWbGQvYsuJ4U2WsfLYp8S3yb4i7OdU7QDn3l_Q9mYis.
- [14] Serguei A. Mokhov, Joey Paquet, and Mourad Debbabi. The use of NLP techniques in static + [17] Serguei A. Mokhov, Joey Paquet, and Mourad Debbabi. The use of NLP techniques in static code analysis to detect weaknesses and vulnerabilities. In Maria Sokolova and Peter van Beek, editors, Proceedings of Canadian Conference on AI’14, volume 8436 of LNAI, pages 326–332. Springer, May 2014. Short paper.
- [15] Parna Niksirat, Adriana Daca, and Krzysztof Skonieczny. The effects of reduced-gravity + [18] Parna Niksirat, Adriana Daca, and Krzysztof Skonieczny. The effects of reduced-gravity on planetary rover mobility. International Journal of Robotics Research, 39(7):797–811, 2020. https://doi.org/10.1177/0278364920913945.
+ + +- [16] Chet Ramey. The Bourne-Again Shell. In Brown and Wilson [4]. + [19] Chet Ramey. The Bourne-Again Shell. In Brown and Wilson [4]. http://aosabook.org/en/bash.html.
- [17] Rob Schreiber. MATLAB. Scholarpedia, 2(6):2929, 2007. + [20] Rob Schreiber. MATLAB. Scholarpedia, 2(6):2929, 2007. http://www.scholarpedia.org/article/MATLAB.
- [18] The MARF Research and Development Group. The Modular Audio Recognition + [21] The MARF Research and Development Group. The Modular Audio Recognition Framework and its Applications. [online], 2002–2014. http://marf.sf.net and http://arxiv.org/abs/0905.1235, last viewed May 2015.
diff --git a/doc/web/speed-manual.css b/doc/web/speed-manual.css index d00465d..6d423e9 100644 --- a/doc/web/speed-manual.css +++ b/doc/web/speed-manual.css @@ -140,6 +140,7 @@ table.pmatrix {width:100%;} span.bar-css {text-decoration:overline;} img.cdots{vertical-align:middle;} .partToc a, .partToc, .likepartToc a, .likepartToc {line-height: 200%; font-weight:bold; font-size:110%;} +.chapterToc a, .chapterToc, .likechapterToc a, .likechapterToc, .appendixToc a, .appendixToc {line-height: 200%; font-weight:bold;} .index-item, .index-subitem, .index-subsubitem {display:block} div.caption {text-indent:-2em; margin-left:3em; margin-right:1em; text-align:left;} div.caption span.id{font-weight: bold; white-space: nowrap; }