Table of Contents
How to use the MBB platform (Montpellier Bioinformatics Biodiversity computing cluster)
fc - 17.3.2014
This documentation summarizes information to help AMAP agents to connect with the ssh protocol on the MBB cluster and launch heavy computations.
AMAP is part of the Labex Cemeb and the AMAP members can access the Labex resources, including the MBB platform.
For questions about the MBB cluster, please have a look to the online documentation (see links below), then ask first the two AMAP referents: Francois de Coligny or Yves Dumont, before finally contacting the MBB team: Rémy Dernat (system), Julien Veyssier (development) or Khalid Belkhir.
The MBB plaform online documentation
For all works using the services of the Montpellier Bioinformatics Biodiversity computing platform, please add this text in your publications:
“[Replace_with_your_project_name] analyses largely benefited from the Montpellier Bioinformatics Biodiversity computing cluster platform.”
How to create a SSH access login on the MBB cluster (AMAP members only)
All AMAP members can ask for a login on the MBB cluster from any computer located in the AMAP buildings in Montpellier (PS1, PS2). For AMAP members located elsewhere, a fixed ip address will be requested by the MBB team.
To get such a SSH login, please contact F. de Coligny, we will do it together (according to the MBB system administrator request).
Reminder
MBB web site: click there > click on Login (top of the page) > Create an account > Tracking system
- requester name: de Coligny
- requester email: coligny@cirad.fr
- choose 'Account creation on MBB cluster (SSH)'
- priority 'High'
- requester first name: Francois
- team: 'AMAP',
- project description and software to use,
- then a title and message for the mail requesting the validation to the MBB administrators:
- user name
- user email (will receive the 'confirmation and further instructions' mail)
- user preferred login proposal for the MBB administrator
- answer the spam prevention questions
- submit the request
You will get a mail response with the instruction to login on the cluster.
ssh coligny@162.38.181.181
You will have a home directory created. Once logged in, you can choose a new password with:
passwd
How to copy files into your home directory
Once logged in, you can use the Linux commands to create directories in your home directory: Linux commands.
[coligny@cluster1 ~]$ mkdir data [coligny@cluster1 ~]$ cd data [coligny@cluster1 data]$
To copy a file with scp from your computer to the data/ directory on the cluster; open a terminal on your own computer and type something like this (adapt for your own file):
coligny@marvin-13:~/workspace/amapstudio/data/opf$ scp sunflower_100.opf coligny@162.38.181.181:data/ Password: sunflower_100.opf 100% 142KB 142.4KB/s 00:00
To copy a data/ directory from your computer to the cluster into test/:
coligny@marvin-13:~/des-trucs$ scp -r data/ coligny@162.38.181.181:test Password: archimed1Export.ops 100% 378 0.4KB/s 00:00 mod_Rauh_14.opf 100% 31MB 10.3MB/s 00:03 mod_Koriba_7.opf 100% 3812KB 3.7MB/s 00:00 archimed1.config 100% 486 0.5KB/s 00:00 two-trees.ops 100% 386 0.4KB/s 00:00
A documentation about scp: (french)
Example 1: how to install AMAPstudio through svn (modellers full local copy) on the cluster in your home directory
Typing the commands below will create a workspace/amapstudio/ directory and get a local copy of AMAPstudio in it by running an svn checkout (AMAPstudio modelers grant level requested).
[coligny@cluster1 ~]$ pwd /home/fdcoligny [coligny@cluster1 ~]$ mkdir workspace [coligny@cluster1 ~]$ cd workspace/ [coligny@cluster1 workspace]$ mkdir amapstudio [coligny@cluster1 workspace]$ cd amapstudio/ [coligny@cluster1 amapstudio]$ [coligny@cluster1 amapstudio]$ svn co --username coligny https://amap-dev.cirad.fr/svn/jeeb/trunk (...) A trunk/etc/simeo.models A trunk/etc/editoGreenPin.opf A trunk/etc/simeo.extension.list A trunk/etc/editoRedPin.gwa A trunk/etc/xplo.models A trunk/Readme.txt U trunk Révision 4393 extraite.
Example 2: how to install AMAPstudio (end-user IzPack installer) on the cluster
1. create an AMAPstudio installer and copy it to the cluster
coligny@marvin-13:~/workspace/amapstudio$ sh ant installer coligny@marvin-13:~/workspace/amapstudio$ scp AMAPstudio-setup.jar coligny@162.38.181.181:test/installer-test
2. run the installer on the cluster with the GUI on your machine (delays will be experienced due to data transfer time)
ssh -f -Y coligny@162.38.181.181 java -jar test/installer-test/AMAPstudio-setup.jar
How to launch a job on the cluster
A documentation for the Sun Grid Engine (SGE) : the distributed resource management software used to launch jobs on the MBB cluster.
See this particular page: Submitting batch jobs explains how the qsub, qdel and qstat commands can be used to manage batch jobs on the cluster.
You may also read the 'Introduction to sge' man page :
man sge_intro
Example 1: how to compile AMAPstudio on the cluster
Note: AMAPstudio must be compiled with java 1.6. The master node on the cluster contains java 1.7, the compute-nodes run java 1.6 (17 March 2014) → the compilaton must be launched on the cluster nodes with qsub to match the good java 1.6.
1. Open a terminal, login to the cluster and change directory to be in the AMAPstudio installation directory (contains simeo.sh)
ssh coligny@162.38.181.181 (...) cd workspace/amapstudio/trunk
2. Prepare an sge script to launch 'sh ant clean compile':
cat > ant-clean-compile.sge
#!/bin/bash # #$ -cwd #$ -j y #$ -S /bin/bash # date sh ant clean compile date
3. Run the sge script with qsub:
[coligny@cluster1 trunk]$ qsub ant-clean-compile.sge
Your job 166092 ("ant-clean-compile.sge") has been submitted
4. Check the cluster statistics
[coligny@cluster1 trunk]$ qstat job-ID prior name user state submit/start at queue slots ja-task-ID ----------------------------------------------------------------------------------------------------------------- (...) 165554 0.55500 PhyML mbb r 03/10/2014 07:52:56 mbb.q@compute-0-15.local 1 165557 0.55500 PhyML mbb r 03/10/2014 11:21:29 mbb.q@compute-0-19.local 1 166069 0.55500 g32407_top igalaxy r 03/17/2014 10:54:46 mbb.q@compute-0-14.local 1 166091 0.55500 macse mbb r 03/17/2014 14:29:32 mbb.q@compute-0-5.local 1 166092 0.00000 ant-clean- coligny qw 03/17/2014 14:30:25 1
5. When the job is over, get the job output
[coligny@cluster1 trunk]$ less ant-clean-compile.sge.o166092
Mon Mar 17 14:30:32 CET 2014
Buildfile: build.xml
clean:
[delete] Deleting directory /home/fdcoligny/workspace/amapstudio/trunk/bin
revision:
[echo] Jeeb revision 4375
[echo] os.name: Linux
[echo] os.arch: amd64
compile:
[mkdir] Created dir: /home/fdcoligny/workspace/amapstudio/trunk/bin
[javac] Compiling 1872 source files to /home/fdcoligny/workspace/amapstudio/trunk/bin
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[copy] Copying 2608 files to /home/fdcoligny/workspace/amapstudio/trunk/bin
[copy] Copied 28 empty directories to 9 empty directories under /home/fdcoligny/workspace/amapstudio/trunk/bin
BUILD SUCCESSFUL
Total time: 24 seconds
Mon Mar 17 14:30:56 CET 2014
The compilation was successful
Example 2: how to run an AMAPstudio script on the cluster
This example shows how to run the script ScriptTristan2012 on the cluster. This script needs two parameters: an 'ops' file name and an 'archimed' config file. All the related files must be copied on the cluster before running the script.
The syntax to launch the script on a simple machine (needs adaptations on the cluster, see below).
sh simeo.sh -p script jeeb.simeo.module.simeoeditor.scripts.ScriptTristan2012 opsFileName archimed1ConfigFileName
1. Copy the parameter files on the cluster
On the cluster, prepare directories to host the parameter files.
[coligny@cluster1 trunk]$ mkdir tristan-files [coligny@cluster1 trunk]$ cd tristan-files/ [coligny@cluster1 tristan-files]$ mkdir opf
From your machine, copy the needed files into these directories.
coligny@marvin-13:~/des-trucs/data$ scp * coligny@162.38.181.181:workspace/amapstudio/trunk/tristan-files Password: archimed1.config 100% 486 0.5KB/s 00:00 opf: not a regular file two-trees.ops 100% 386 0.4KB/s 00:00 coligny@marvin-13:~/des-trucs/data$ scp opf/* coligny@162.38.181.181:workspace/amapstudio/trunk/tristan-files/opf Password: mod_Koriba_7.opf 100% 2321KB 2.3MB/s 00:00 mod_Rauh_14.opf
2. Prepare a sge script to launch the Script (in the amapstudio install dir)
script-Tristan-2012.sge
#!/bin/bash # #$ -cwd #$ -j y #$ -S /bin/bash # date sh simeo.sh -p script jeeb.simeo.module.simeoeditor.scripts.ScriptTristan2012 tristan-files/two-trees.ops tristan-files/archimed1.config date
3. Run the script
[coligny@cluster1 trunk]$ qsub script-Tristan-2012.sge
Your job 166129 ("script-Tristan-2012.sge") has been submitted
[coligny@cluster1 tristan-files]$ qstat
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
(...)
166129 0.00000 script-Tri coligny qw 03/17/2014 15:41:10 1
4. When the job is over, get the output
[coligny@cluster1 trunk]$ less script-Tristan-2012.sge.o166129 Mon Mar 17 15:41:17 CET 2014 64 bits architecture - max memory: 4096 mega bytes Default locale is: en Simeo root directory: /home/fdcoligny/workspace/amapstudio/trunk Simeo 0.9, (c) 2006-2014 F. de Coligny et al. This program comes with ABSOLUTELY NO WARRANTY; This is free software, and you are welcome to redistribute it under certain conditions; See COPYING and COPYING.LESSER for details. Simeo 0.9-4375 with pilot capsis.script.Pilot: correct boot at 17 Mar 2014 15:41:19 CET Launching script jeeb.simeo.module.simeoeditor.scripts.ScriptTristan2012... ScriptTristan2012... opsFileName : tristan-files/two-trees.ops archimed1ConfigFileName: tristan-files/archimed1.config Loading scene... EditoInitialParameters buildInitScene ()... initXExtension: 50.0 initYExtension: 50.0 -> Creating a cache for scene +1 in *overwrite mode* (overwrite original files, no copies to automatic directories) EditoModel createScene () addingMockup: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf... Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf... Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf done EditoModel createScene () addingMockup: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Koriba_7.opf Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Koriba_7.opf... Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Koriba_7.opf... Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Koriba_7.opf done Initialization completed Memorizer capsis.extension.memorizer.DefaultMemorizer was correctly set for project Project_edito.p Checking first mockup... Mockup id: 1 Mockup a_filePath_0: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf Mockup a_filePath_1: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf Mockup fileName_1: mod_Rauh_14.opf Mockup opf: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf Mockup file: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf Archimed1ExportScript starting... configFileName: tristan-files/archimed1.config #mockups : 2 Archimed1ExportConfigReader found 8 parameters in tristan-files/archimed1.config record: radiativeModel = TranslucentModel:0 record: yToNorthAngle_CW_deg = 45 record: latitude_deg = 45 record: clearness = 0.5 record: time1 = 2012/1/1 record: time2 = 2012/31/24 record: numberOfPixels = 25000 (...)