User Tools

Site Tools


private:howtousethembbcluster

How to use the MBB platform (Montpellier Bioinformatics Biodiversity computing cluster)

fc - 17.3.2014

This documentation summarizes information to help AMAP agents to connect with the ssh protocol on the MBB cluster and launch heavy computations.

AMAP is part of the Labex Cemeb and the AMAP members can access the Labex resources, including the MBB platform.

For questions about the MBB cluster, please have a look to the online documentation (see links below), then ask first the two AMAP referents: Francois de Coligny or Yves Dumont, before finally contacting the MBB team: Rémy Dernat (system), Julien Veyssier (development) or Khalid Belkhir.

The MBB plaform online documentation

For all works using the services of the Montpellier Bioinformatics Biodiversity computing platform, please add this text in your publications:

“[Replace_with_your_project_name] analyses largely benefited from the Montpellier Bioinformatics Biodiversity computing cluster platform.”

How to create a SSH access login on the MBB cluster (AMAP members only)

All AMAP members can ask for a login on the MBB cluster from any computer located in the AMAP buildings in Montpellier (PS1, PS2). For AMAP members located elsewhere, a fixed ip address will be requested by the MBB team.

To get such a SSH login, please contact F. de Coligny, we will do it together (according to the MBB system administrator request).

Reminder

MBB web site: click there > click on Login (top of the page) > Create an account > Tracking system

  • requester name: de Coligny
  • requester email: coligny@cirad.fr
  • choose 'Account creation on MBB cluster (SSH)'
  • priority 'High'
  • requester first name: Francois
  • team: 'AMAP',
  • project description and software to use,
  • then a title and message for the mail requesting the validation to the MBB administrators:
    • user name
    • user email (will receive the 'confirmation and further instructions' mail)
    • user preferred login proposal for the MBB administrator
  • answer the spam prevention questions
  • submit the request

You will get a mail response with the instruction to login on the cluster.

ssh coligny@162.38.181.181

You will have a home directory created. Once logged in, you can choose a new password with:

passwd

How to copy files into your home directory

Once logged in, you can use the Linux commands to create directories in your home directory: Linux commands.

[coligny@cluster1 ~]$ mkdir data
[coligny@cluster1 ~]$ cd data
[coligny@cluster1 data]$ 

To copy a file with scp from your computer to the data/ directory on the cluster; open a terminal on your own computer and type something like this (adapt for your own file):

coligny@marvin-13:~/workspace/amapstudio/data/opf$ scp sunflower_100.opf coligny@162.38.181.181:data/
Password: 
sunflower_100.opf                100%  142KB 142.4KB/s   00:00  

To copy a data/ directory from your computer to the cluster into test/:

coligny@marvin-13:~/des-trucs$ scp -r data/ coligny@162.38.181.181:test
Password: 
archimed1Export.ops                        100%  378     0.4KB/s   00:00    
mod_Rauh_14.opf                            100%   31MB  10.3MB/s   00:03    
mod_Koriba_7.opf                           100% 3812KB   3.7MB/s   00:00    
archimed1.config                           100%  486     0.5KB/s   00:00    
two-trees.ops                              100%  386     0.4KB/s   00:00    

A documentation about scp: (french)

Example 1: how to install AMAPstudio through svn (modellers full local copy) on the cluster in your home directory

Typing the commands below will create a workspace/amapstudio/ directory and get a local copy of AMAPstudio in it by running an svn checkout (AMAPstudio modelers grant level requested).

[coligny@cluster1 ~]$ pwd
/home/fdcoligny

[coligny@cluster1 ~]$ mkdir workspace

[coligny@cluster1 ~]$ cd workspace/

[coligny@cluster1 workspace]$ mkdir amapstudio

[coligny@cluster1 workspace]$ cd amapstudio/
[coligny@cluster1 amapstudio]$ 

[coligny@cluster1 amapstudio]$ svn co --username coligny https://amap-dev.cirad.fr/svn/jeeb/trunk

(...)
A    trunk/etc/simeo.models
A    trunk/etc/editoGreenPin.opf
A    trunk/etc/simeo.extension.list
A    trunk/etc/editoRedPin.gwa
A    trunk/etc/xplo.models
A    trunk/Readme.txt
 U   trunk
Révision 4393 extraite.

Example 2: how to install AMAPstudio (end-user IzPack installer) on the cluster

1. create an AMAPstudio installer and copy it to the cluster

coligny@marvin-13:~/workspace/amapstudio$ sh ant installer

coligny@marvin-13:~/workspace/amapstudio$ scp AMAPstudio-setup.jar coligny@162.38.181.181:test/installer-test

2. run the installer on the cluster with the GUI on your machine (delays will be experienced due to data transfer time)

ssh -f -Y coligny@162.38.181.181 java -jar test/installer-test/AMAPstudio-setup.jar

How to launch a job on the cluster

A documentation for the Sun Grid Engine (SGE) : the distributed resource management software used to launch jobs on the MBB cluster.

See this particular page: Submitting batch jobs explains how the qsub, qdel and qstat commands can be used to manage batch jobs on the cluster.

You may also read the 'Introduction to sge' man page :

man sge_intro

Example 1: how to compile AMAPstudio on the cluster

Note: AMAPstudio must be compiled with java 1.6. The master node on the cluster contains java 1.7, the compute-nodes run java 1.6 (17 March 2014) → the compilaton must be launched on the cluster nodes with qsub to match the good java 1.6.

1. Open a terminal, login to the cluster and change directory to be in the AMAPstudio installation directory (contains simeo.sh)

ssh coligny@162.38.181.181
(...)

cd workspace/amapstudio/trunk

2. Prepare an sge script to launch 'sh ant clean compile':

cat > ant-clean-compile.sge

#!/bin/bash
#
#$ -cwd
#$ -j y
#$ -S /bin/bash
#
date
sh ant clean compile
date

3. Run the sge script with qsub:

[coligny@cluster1 trunk]$ qsub ant-clean-compile.sge
Your job 166092 ("ant-clean-compile.sge") has been submitted

4. Check the cluster statistics

[coligny@cluster1 trunk]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
(...)
165554 0.55500 PhyML      mbb          r     03/10/2014 07:52:56 mbb.q@compute-0-15.local           1        
165557 0.55500 PhyML      mbb          r     03/10/2014 11:21:29 mbb.q@compute-0-19.local           1        
166069 0.55500 g32407_top igalaxy      r     03/17/2014 10:54:46 mbb.q@compute-0-14.local           1        
166091 0.55500 macse      mbb          r     03/17/2014 14:29:32 mbb.q@compute-0-5.local            1        
166092 0.00000 ant-clean- coligny      qw    03/17/2014 14:30:25                                    1        

5. When the job is over, get the job output

[coligny@cluster1 trunk]$ less ant-clean-compile.sge.o166092

Mon Mar 17 14:30:32 CET 2014
Buildfile: build.xml

clean:
   [delete] Deleting directory /home/fdcoligny/workspace/amapstudio/trunk/bin

revision:
     [echo] Jeeb revision 4375
     [echo] os.name: Linux
     [echo] os.arch: amd64

compile:
    [mkdir] Created dir: /home/fdcoligny/workspace/amapstudio/trunk/bin
    [javac] Compiling 1872 source files to /home/fdcoligny/workspace/amapstudio/trunk/bin
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
     [copy] Copying 2608 files to /home/fdcoligny/workspace/amapstudio/trunk/bin
     [copy] Copied 28 empty directories to 9 empty directories under /home/fdcoligny/workspace/amapstudio/trunk/bin

BUILD SUCCESSFUL
Total time: 24 seconds
Mon Mar 17 14:30:56 CET 2014

The compilation was successful

Example 2: how to run an AMAPstudio script on the cluster

This example shows how to run the script ScriptTristan2012 on the cluster. This script needs two parameters: an 'ops' file name and an 'archimed' config file. All the related files must be copied on the cluster before running the script.

The syntax to launch the script on a simple machine (needs adaptations on the cluster, see below).

sh simeo.sh -p script jeeb.simeo.module.simeoeditor.scripts.ScriptTristan2012 opsFileName archimed1ConfigFileName

1. Copy the parameter files on the cluster

On the cluster, prepare directories to host the parameter files.

[coligny@cluster1 trunk]$ mkdir tristan-files
[coligny@cluster1 trunk]$ cd tristan-files/
[coligny@cluster1 tristan-files]$ mkdir opf

From your machine, copy the needed files into these directories.

coligny@marvin-13:~/des-trucs/data$ scp * coligny@162.38.181.181:workspace/amapstudio/trunk/tristan-files
Password: 
archimed1.config                      100%  486     0.5KB/s   00:00    
opf: not a regular file
two-trees.ops                         100%  386     0.4KB/s   00:00    

coligny@marvin-13:~/des-trucs/data$ scp opf/* coligny@162.38.181.181:workspace/amapstudio/trunk/tristan-files/opf
Password: 
mod_Koriba_7.opf                      100% 2321KB   2.3MB/s   00:00    
mod_Rauh_14.opf           

2. Prepare a sge script to launch the Script (in the amapstudio install dir)

script-Tristan-2012.sge

#!/bin/bash
#
#$ -cwd
#$ -j y
#$ -S /bin/bash
#
date
sh simeo.sh -p script jeeb.simeo.module.simeoeditor.scripts.ScriptTristan2012 tristan-files/two-trees.ops tristan-files/archimed1.config
date

3. Run the script

[coligny@cluster1 trunk]$ qsub script-Tristan-2012.sge 
Your job 166129 ("script-Tristan-2012.sge") has been submitted

[coligny@cluster1 tristan-files]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
(...)
166129 0.00000 script-Tri coligny      qw    03/17/2014 15:41:10                                    1   

4. When the job is over, get the output

[coligny@cluster1 trunk]$ less script-Tristan-2012.sge.o166129 

Mon Mar 17 15:41:17 CET 2014
64 bits architecture - max memory: 4096 mega bytes
Default locale is: en 
Simeo root directory: /home/fdcoligny/workspace/amapstudio/trunk
Simeo 0.9, (c) 2006-2014 F. de Coligny et al.
This program comes with ABSOLUTELY NO WARRANTY;
This is free software, and you are welcome to redistribute it
under certain conditions; See COPYING and COPYING.LESSER for details.

Simeo 0.9-4375 with pilot capsis.script.Pilot: correct boot at 17 Mar 2014 15:41:19 CET

Launching script jeeb.simeo.module.simeoeditor.scripts.ScriptTristan2012...
ScriptTristan2012...
opsFileName            : tristan-files/two-trees.ops
archimed1ConfigFileName: tristan-files/archimed1.config
Loading scene... 
EditoInitialParameters buildInitScene ()...
  initXExtension: 50.0
  initYExtension: 50.0
-> Creating a cache for scene +1 in *overwrite mode* (overwrite original files, no copies to automatic directories)
EditoModel createScene () addingMockup: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf
Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf...
Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf...
Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf done
EditoModel createScene () addingMockup: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Koriba_7.opf
Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Koriba_7.opf...
Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Koriba_7.opf...
Loading /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Koriba_7.opf done
Initialization completed
Memorizer capsis.extension.memorizer.DefaultMemorizer was correctly set for project Project_edito.p
Checking first mockup...
  Mockup id: 1
  Mockup a_filePath_0: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf
  Mockup a_filePath_1: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf
  Mockup   fileName_1: mod_Rauh_14.opf
  Mockup opf: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf
  Mockup file: /home/fdcoligny/workspace/amapstudio/trunk/tristan-files/opf/mod_Rauh_14.opf
Archimed1ExportScript starting...
configFileName: tristan-files/archimed1.config
#mockups      : 2
Archimed1ExportConfigReader found 8 parameters in tristan-files/archimed1.config
record: radiativeModel = TranslucentModel:0
record: yToNorthAngle_CW_deg = 45
record: latitude_deg = 45
record: clearness = 0.5
record: time1 = 2012/1/1
record: time2 = 2012/31/24
record: numberOfPixels = 25000

(...)
private/howtousethembbcluster.txt · Last modified: 2021/12/17 09:22 by 127.0.0.1