Genome Analysis Wiki - User contributions [en]

GotCloud: Binary Releases

2013-10-14T13:11:11Z

Terry Gliedt: /* Install the GotCloud Debian Packages */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Installing Debian packages normally requires root access, however, you can install as a non-root user (see below).

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian Packages====

Install Debian packages with root access

Installation normally requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================

sudo dpkg -i ~/debs/gotcloud-test_*_amd64.deb
</code>

Install Debian packages without root access

Installing Debian packages generally requires root access, however, you can install as a non-root user using the following instructions (bash commands shown here):
<code>
tempdir=/tmp/$$.gotcloud.removeme # Temp directory
destdir=$HOME # Where you want GotCloud installed

dpkg -x [path_to_deb_file]/gotcloud-bin_*_amd64.deb $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

dpkg -x [path_to_deb_file]/gotcloud-test_*_amd64.deb $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

$destdir/gotcloud/scripts/check_requirements.sh # Check requirements
</code>

If there were messages about missing packages, install them with the commands shown by the script. Installing missing packages will require root access.

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8 # For example
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud: Binary Releases

2013-10-14T13:10:56Z

Terry Gliedt: /* Install the GotCloud Debian Packages */

GotCloud: Binary Releases

2013-10-14T13:10:14Z

Terry Gliedt: /* Install the GotCloud Debian bin Package */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Installing Debian packages normally requires root access, however, you can install as a non-root user (see below).

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian Packages====

Install Debian packages with root access

Installation normally requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

Install Debian packages without root access

Installing Debian packages generally requires root access, however, you can install as a non-root user using the following instructions (bash commands shown here):
<code>
tempdir=/tmp/$$.gotcloud.removeme # Temp directory
destdir=$HOME # Where you want GotCloud installed

dpkg -x [path_to_deb_file]/gotcloud-bin_*_amd64.deb $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

dpkg -x [path_to_deb_file]/gotcloud-test_*_amd64.deb $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

$destdir/gotcloud/scripts/check_requirements.sh # Check requirements
</code>

If there were messages about missing packages, install them with the commands shown by the script. Installing missing packages will require root access.

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8 # For example
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud: Binary Releases

2013-10-14T13:09:48Z

Terry Gliedt: /* Install the GotCloud Debian bin Package */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Installing Debian packages normally requires root access, however, you can install as a non-root user (see below).

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian bin Package====

Install Debian packages with root access

Installation normally requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

Install Debian packages without root access

Installing Debian packages generally requires root access, however, you can install as a non-root user using the following instructions (bash commands shown here):
<code>
tempdir=/tmp/$$.gotcloud.removeme # Temp directory
destdir=$HOME # Where you want GotCloud installed

dpkg -x [path_to_deb_file]/gotcloud-bin_*_amd64.deb $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

dpkg -x [path_to_deb_file]/gotcloud-test_*_amd64.deb $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

$destdir/gotcloud/scripts/check_requirements.sh # Check requirements
</code>

If there were messages about missing packages, install them with the commands shown by the script. Installing missing packages will require root access.

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8 # For example
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud: Binary Releases

2013-10-14T13:09:02Z

Terry Gliedt: /* Install the GotCloud Debian bin Package */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Installing Debian packages normally requires root access, however, you can install as a non-root user (see below).

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian bin Package====

Installation normally requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

Install Debian packages without root access
Installing Debian packages generally requires root access, however, you can install as a non-root user using the following instructions (bash commands shown here):
<code>
tempdir=/tmp/$$.gotcloud.removeme # Temp directory
destdir=$HOME # Where you want GotCloud installed

dpkg -x [path_to_deb_file]/gotcloud-bin_*_amd64.deb $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

dpkg -x [path_to_deb_file]/gotcloud-test_*_amd64.deb $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

$destdir/gotcloud/scripts/check_requirements.sh # Check requirements
</code>

If there were messages about missing packages, install them with the commands shown by the script. Installing missing packages will require root access.

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8 # For example
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud: Binary Releases

2013-10-14T13:07:55Z

Terry Gliedt: /* Install the GotCloud Debian bin Package */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Installing Debian packages normally requires root access, however, you can install as a non-root user (see below).

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian bin Package====

Installation normally requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

Install Debian packages without root access
Installing Debian packages generally requires root access, however, you can install as a non-root user using the following instructions (bash commands shown here):
<code>
tempdir=/tmp/$$.gotcloud.removeme # Temp directory
destdir=$HOME # Where you want GotCloud installed

pkg=[path_to_deb_file]/gotcloud-bin_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

pkg=[path_to_deb_file]/gotcloud-test_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

$destdir/gotcloud/scripts/check_requirements.sh # Check requirements
</code>

If there were messages about missing packages, install them with the commands shown by the script. Installing missing packages will require root access.

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8 # For example
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud: Binary Releases

2013-10-14T13:07:12Z

Terry Gliedt: /* Install the GotCloud Debian bin Package */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Installing Debian packages normally requires root access, however, you can install as a non-root user (see below).

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian bin Package====

Installation normally requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

Install Debian packages without root access
Installing Debian packages generally requires root access, however, you can install as a non-root user using the following instructions (bash commands shown here):
<code>
tempdir=/tmp/$$.gotcloud.removeme # Temp directory
destdir=$HOME # Where you want GotCloud installed

pkg=[path_to_deb_file]/gotcloud-bin_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

pkg=[path_to_deb_file]/gotcloud-test_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

$destdir/gotcloud/scripts/check_requirements.sh # Check requirements
</code>

If there were messages about missing packages, install them with the commands shown by the script:

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8 # For example
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud: Binary Releases

2013-10-14T13:06:36Z

Terry Gliedt: /* Install the GotCloud Debian bin Package */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Installing Debian packages normally requires root access, however, you can install as a non-root user (see below).

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian bin Package====

Installation normally requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

Install Debian packages without root access
Installing Debian packages generally requires root access, however, you can install as a non-root user using the following instructions (bash commands shown here):
<code>
tempdir=/tmp/$$.gotcloud.removeme # Temp directory
destdir=$HOME # Where you want GotCloud installed

pkg=[path_to_deb_file]/gotcloud-bin_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

pkg=[path_to_deb_file]/gotcloud-test_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

$destdir/gotcloud/scripts/check_requirements.sh # Check requirements
</code>

If there were messages about missing packages, install them with the commands provided by the script:

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8 # For example
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud: Binary Releases

2013-10-14T13:05:07Z

Terry Gliedt: /* Installing Debian Packages */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Installing Debian packages normally requires root access, however, you can install as a non-root user (see below).

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian bin Package====

Installation normally requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

If there were messages about missing packages, install them with the commands provided by the script:

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8 # For example
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud: Binary Releases

2013-10-14T13:00:42Z

Terry Gliedt: /* Installing Debian Packages */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Install Debian packages as root
<code>
sudo dpkg -i [path_to_deb_file]/gotcloud-bin_*_amd64.deb
sudo dpkg -i [path_to_deb_file]/gotcloud-test_*_amd64.deb
</code>

Install Debian packages without root access
Installing Debian packages generally requires root access, however, you can install as a non-root user using the following instructions (bash commands shown here):
<code>
tempdir=/tmp/$$.gotcloud.removeme # Temp directory
destdir=$HOME # Where you want GotCloud installed

pkg=[path_to_deb_file]/gotcloud-bin_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

pkg=[path_to_deb_file]/gotcloud-test_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

$destdir/gotcloud/scripts/check_requirements.sh # Check requirements
</code>

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian bin Package====

Installation requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

If there were messages about missing packages, install them now

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud: Binary Releases

2013-10-14T12:57:24Z

Terry Gliedt: /* Installing Debian Packages */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Installing Debian packages generally requires root access, however, you can install as a non-root user using the following instructions (bash commands shown here):
<code>
tempdir=/tmp/$$.gotcloud.removeme # Temp directory
destdir=$HOME # Where you want GotCloud installed

pkg=[path_to_deb_file]/gotcloud-bin_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

pkg=[path_to_deb_file]/gotcloud-test_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

$destdir/gotcloud/scripts/check_requirements.sh # Check requirements
</code>

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian bin Package====

Installation requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

If there were messages about missing packages, install them now

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud: Binary Releases

2013-10-14T12:56:46Z

Terry Gliedt: /* GotCloud Debian Packages */

Back to parent: [[GotCloud]]

The [[GotCloud]] software has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested. It is not supported on Windows.

If you run into any problems with GotCloud, please check: [[GotCloud: FAQs]] or for problems/solutions for a specific version, [[GotCloud:_FAQs#Version_Problems|Version Problems]].

=GotCloud Binary Releases=
GotCloud releases with the precompiled binary executables are available for download.

The binaries were compiled on Linux Ubuntu.

If you need the source, see [[GotCloud: Source Releases]] for information on obtaining the source.

The Binary Releases are available as:
* [[#Debian Packages|debian packages]] - require root access to install
* [[#Tar Binary Releases|tarballs]] - anyone can untar

The GotCloud software is distributed in several packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''.

There is also a separate package to test the installation.

==GotCloud Debian Packages==

If you would like to install from:
* tar'd binary packages, see: [[#Tar Binary Releases|Tar Binary Releases]]
* source (does not require root access), see: [[GotCloud: Source Releases]]

=== Installing Debian Packages ===

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

Installing Debian packages generally requires root access, however, you can install as a non-root user using the following instructions (bash commands shown here):
<code>
tempdir=/tmp/$$.gotcloud.removeme # Temp directory
destdir=$HOME # Where you want GotCloud installed

pkg=[path_to_deb_file]/gotcloud-bin_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

pkg=[path_to_deb_file]/gotcloud-test_*_amd64.deb
dpkg -x $pkg $tempdir
rsync -av --delete $tempdir/usr/local/gotcloud/* $destdir/gotcloud || exit 1
rm -rf $tempdir

$destdir/gotcloud/scripts/check_requirements.sh # Check requirements
</code>

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the GotCloud Debian Packages====

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

====Install the GotCloud Debian bin Package====

Installation requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

If there were messages about missing packages, install them now

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (not as root) and do:

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

== Tar Binary Releases ==

The tar binary releases contain the precompiled binaries and scripts necessary to run gotcloud.

If you would like to install from:
* debian packages, see: [[#GotCloud Debian Packages|Debian Packages]]
* source, see: [[GotCloud: Source Releases]]

=== Installing Tar Releases ===

The tar files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloudBin_latest.tgz aligner and variant caller
gotcloudTest_latest.tgz test data for software
M.n/gotcloud-bin_M.n.tar.gz version M.n of aligner and variant caller
M.n/gotcloud-test_M.n.tar.gz version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

====Fetch the Tar Releases====

<code>
# The easiest way:
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud-\*latest\*.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud-*latest*.tgz
ftp> quit
221 Goodbye.
</code>

====Unpack the Tar Release====

Unpack the realeases:
tar xf gotcloud-*latest*.tgz

===Testing the GotCloud Release===

We recommend that at least the first time, you download the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

The test case results are self-checked and if errors should occur, it will be obvious.

<code>
# Test the aligner (3 minutes +/- 1 minute)
'''gotcloud/bin/gotcloud align --test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and 'gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (7 minutes +/-2 minute)
'''gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and 'gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

=Creating Your Own Debian Package From the Source=

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

GotCloud

2013-10-14T12:45:02Z

Terry Gliedt: /* GotCloud Dependencies */

= Genomes on the Cloud (GotCloud) Introduction=

To handle the increasing volume of next generation sequencing and genotyping data, we created and developed software pipelines called '''Genomes on the Cloud (GotCloud).'''

GotCloud contains Mapping & Variant Calling Pipelines.

Key Features:
* Connects sequence analysis tools together in automated pipeline
** Alignment, quality control, variant calling
* Robust against unexpected system failure using GNU make
** easy restart after failure
* Massively parallel, can run hundreds of jobs
** Splits large jobs into many pieces
** Simplifies running on clusters
* Scalable to tens of thousands of samples
* Easy to use - Automates series of configurable steps
** user doesn't have to understand/configure/know the many tools required to create high quality results
* Available on Amazon Web Services (AWS) Elastic Compute Cloud (EC2)
* Run on local machines/clusters
* Available via Debian Packages

GotCloud incorporates the alignment and variant calling pipelines that we have been using at UM into one easy to use, publicly available tool. GotCloud can run on a user's computer, on an instance in a
compute cloud, and/or can split the work up onto a cluster of machines or instances.

== Getting Help with GotCloud ==

Please join in the [http://groups.google.com/group/GotCloud GotCloud Google Group] to ask / discuss / comment about these pipelines.

Currently the "join" button appears to be missing. Click "NEW TOPIC", then select "Join this group". You can then cancel the message post (or post a message).

See [[GotCloud: FAQs]] if you have any questions. If your questions are not answered there, join the google group and ask, or email Mary Kate Wing (mktrost@umich.edu).

== Sequence Analysis Background Information ==

There are many essential steps in the analysis of next generation sequence data.

Next generation sequence data analysis starts with [http://en.wikipedia.org/wiki/FASTQ_format FASTQ files], the typical format provided from your sequencing center containing the sequence & base quality information for your data.

The fastq files are processed using the alignment pipeline which finds the most likely genomic location for each read and stores that information in a [[BAM|BAM (Binary Sequence Alignment/Map format) file]]. In addition to the sequence and base quality information contained in FASTQ files, a BAM file also contains the genomic location and some additional information about the mapping. As part of the alignment pipeline, the base qualities are adjusted to more accurately reflect the likelihood that the base is correct.

The [[GotCloud: Alignment Pipeline|alignment pipeline]] can be skipped if you already have Deduped and Recalibrated BAM files.

The [[GotCloud: Variant Calling Pipeline|variant calling pipeline]] processes the deduped and recalibrated BAMs file produced by the alignment pipeline or that you provide it, generating an initial list of polymorphic sites and genotypes stored in a [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) file]. The variant calling pipeline then filters the variants using both hard filters and a [[SVM Filtering|Support Vector Machine (SVM)]]. It then uses haplotype information to refine these genotypes in an updated VCF file.

After completing the GotCloud Variant Calling Pipeline, [[EPACTS|EPACTS (Efficient and Parallelizable Association Container Toolbox)]] can be used to perform statistical tests to identify genome-wide association from sequence data.

[[File:GotCloudDiagram.jpg|500px]]

= GotCloud Setup =

You may run the GotCloud software in several modes:
* On your own hardware running Ubuntu or Redhat/CentOS. See the instructions about installing the software below.
* On an Amazon Elastic Compute Cloud using Ubuntu or Redhat/CentOS if you do not have your own set of machines.
** See [[GotCloud: Amazon]] for more information.
** You can run on an EC2 cluster instance created by StarCluster.

GotCloud has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested.

== GotCloud Dependencies ==

GotCloud requires certain things to be installed in order to run:
* perl - gotcloud is a perl script and it calls many other perl scripts
** Zlib.pm - required for perl scripts to read compressed files.
* make - GNU make is used to run the pipelines
* java - required to run the beagle step of the ld-aware genotype refinement

You can check if your system has the proper software installed by invoking the command
[gotcloud_path]/scripts/check_requirements.sh

== Install GotCloud Software ==

You can install gotCloud on your system as (follow the links for the appropriate instructions):
* [[GotCloud: Source Releases|source release]] - contains the scripts and uncompiled source
* [[GotCloud: Binary Releases|binary release]] - contains the scripts and pre-compiled binaries (no source)

[[GotCloud: Versions]] describes the changes added to each version.

Alternatively if you are using Amazon EC2, you can use the following instructions:
* Create a machine instance based on the AMI we provide. ''Instructions coming soon''
** Directions for how we create an AMI: [[Creating an AMI on EC2]]
'''or''' Create an EC2 cluster instance using [[StarCluster|StarCluster]]
* Prepare your data for processing by GotCloud [[AWS Resources|AWS Resources]]

For more information on Amazon Web Services, see: https://aws.amazon.com/

== GotCloud Reference/Resource Files ==
In order to run gotCloud, you need to provide Genetic Reference and Resource Files.

These include information about the reference sequence and dbnsp positions.

See: [[GotCloud: Genetic Reference and Resource Files]] for information about the required files. It contains a description of the required files, information about generating your own versions, as well as a downloadable set of files.

== Configure GotCloud ==
* [[Configure GotCloud|Configure Gotcloud]] for your installation

== Running GotCloud Software ==

* [[GotCloud: Alignment Pipeline|Alignment Pipeline]]
* [[GotCloud: Variant Calling Pipeline|Variant Calling Pipeline]]

[[Tutorial: GotCloud]]

== Development Notes ==
* [[Creating Packages]]
* Amazon EC2
** [[Creating a Snapshot on EC2]]
** [[Creating an AMI on EC2]]
** [[Mount S3 Volume]]
** Notes on sequence data preparation in [[Amazon Storage|Amazon Storage]].

GotCloud: Source Releases

2013-10-14T12:40:22Z

Terry Gliedt: /* Test snp calling */

GotCloud: Source Releases

2013-10-14T12:39:59Z

Terry Gliedt: /* Test the aligner */

GotCloud: Source Releases

2013-10-14T12:39:30Z

Terry Gliedt: /* Test the aligner */

GotCloud: Source Releases

2013-10-14T12:35:19Z

Terry Gliedt: /* Building */

Creating an AMI on EC2

2013-06-17T17:10:13Z

Terry Gliedt: /* Configure the Host to be Usable */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=524288 count=16384 # 8GB swap on t1.micro 15G=bs=1073741824 count=15
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, if it is set up properly.
Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Configure s3cmd. This will ask for your AWS ID and Secret Key. If creates a file in ~/.s3cfg

<code>
s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: password_you_do_not_need_to_know
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf
Encryption password: password_you_do_not_need_to_know
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/home/ubuntu/.s3cfg'
</code>

'''(4)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=/etc/passwd-s3fs # Where s3fs access info will live
S3ERR=/tmp/s3fs.err
# These are needed for s3fs access
AWSACCESSKEYID=AKIAxxxxxxZ3YCZF2Q
AWSSECRETACCESSKEY=ft1eJa3WxxxxxxxNlbA08x/G8iMqkMIkJjFCIGf

# Check that we have swap set up
a=`swapon -s | grep -v File`
if [ "$a" = "" ]; then
echo "#######################################################"
echo "# You have no SWAP file set up"
echo ""
echo "# swap=/mnt/swapfile"
echo "# sudo dd if=/dev/zero of=$swap bs=1073741824 count=20"
echo "# sudo chown root:root $swap"
echo "# sudo mkswap $swap"
echo "# sudo chmod 0600 $swap"
echo "# sudo swapon $swap"
echo ""
echo "# If need be, add to /etc/fstab"
echo "# echo "$swap none swap sw 0 0" >> /etc/fstab"
echo "#######################################################"
fi

# Set up for GotCloud
gc=/gotcloud.mnt
if [ ! -r $gc/release_version.txt ]; then
mkdir -p $gc
mount /dev/xvdg $gc
if [ -d $gc/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is set up on $gc"
echo "#######################################################"
fi
fi

# Set up access to S3 storage as normal filesystem
echo "${AWSACCESSKEYID}:$AWSSECRETACCESSKEY" > $FILES3
chown root.root $FILES3
chmod 640 $FILES3

usermod -aG fuse $USER

# Setup 1000genomes
mkdir -p $THOUSANDG
if [ ! -r $THOUSANDG/release ]; then
chown $USER.$USER $THOUSANDG
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################"
echo "# 1000genomes is not set up on $THOUSANDG"
echo "# See S3FS errors in $S3ERR"
echo "#######################################################"
fi
df -h
fi
exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name GotCLoud 1.06
Image Description: From CSG at University of Michigan
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or larger -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-06-17T17:08:56Z

Terry Gliedt: /* Configure the Host to be Usable */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=524288 count=16384 # 8GB swap on t1.micro 15G=bs=1073741824 count=15
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, if it is set up properly.
Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Configure s3cmd. This will ask for your AWS ID and Secret Key. If creates a file in ~/.s3cfg

<code>
s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: password_you_do_not_need_to_know
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf
Encryption password: password_you_do_not_need_to_know
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/home/ubuntu/.s3cfg'
</code>

'''(4)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=/etc/passwd-s3fs # Where s3fs access info will live
S3ERR=/tmp/s3fs.err
# These are needed for s3fs access
AWSACCESSKEYID=AKIAxxxxxxZ3YCZF2Q
AWSSECRETACCESSKEY=ft1eJa3WxxxxxxxNlbA08x/G8iMqkMIkJjFCIGf

# Check that we have swap set up
a=`swapon -s | grep -v File`
if [ "$a" = "" ]; then
echo "#######################################################"
echo "# You have no SWAP file set up"
echo ""
echo "# swap=/mnt/swapfile"
echo "# sudo dd if=/dev/zero of=$swap bs=1073741824 count=20"
echo "# sudo chown root:root $swap"
echo "# sudo mkswap $swap"
echo "# sudo chmod 0600 $swap"
echo "# sudo swapon $swap"
echo ""
echo "# If need be, add to /etc/fstab"
echo "# echo "$swap none swap sw 0 0" >> /etc/fstab"
echo "#######################################################"
fi

# Set up for GotCloud
gc=/gotcloud.mnt
if [ ! -r $gc/release_version.txt ]; then
mkdir -p $gc
mount /dev/xvdg $gc
if [ -d $gc/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is set up on $gc"
echo "#######################################################"
fi
fi

# Set up access to S3 storage as normal filesystem
echo "${AWSACCESSKEYID}:$AWSSECRETACCESSKEY" > $FILES3
chown root.root $FILES3
chmod 640 $FILES3

usermod -aG fuse $USER

# Setup 1000genomes
mkdir -p $THOUSANDG
if [ ! -r $THOUSANDG/release ]; then
chown $USER.$USER $THOUSANDG
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################"
echo "# 1000genomes is not set up on $THOUSANDG"
echo "# See S3FS errors in $S3ERR"
echo "#######################################################"
fi
df -h
fi
exit 0

</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name GotCLoud 1.06
Image Description: From CSG at University of Michigan
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or larger -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-06-17T17:06:07Z

Terry Gliedt: /* Set Up Swap Space */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=524288 count=16384 # 8GB swap on t1.micro 15G=bs=1073741824 count=15
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, if it is set up properly.
Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Configure s3cmd. This will ask for your AWS ID and Secret Key. If creates a file in ~/.s3cfg

<code>
s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: password_you_do_not_need_to_know
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf
Encryption password: password_you_do_not_need_to_know
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/home/ubuntu/.s3cfg'
</code>

'''(4)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

# Make sure we have a swap file
a=`swapon -s | grep -v Filename'
if [ "$a" = "" ]; then
echo "#######################################################"
echo "# You have no SWAP file set up"
echo "#"
echo "# swap=/mnt/swapfile"
echo "# sudo dd if=/dev/zero of=$swap bs=1073741824 count=20"
echo "# sudo chown root:root $swap"
echo "# sudo mkswap $swap"
echo "# sudo chmod 0600 $swap"
echo "# sudo swapon $swap"
echo "#"
echo "# If need be, add to /etc/fstab"
echo "# echo "$swap none swap sw 0 0" >> /etc/fstab"
echo "#######################################################"
fi

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name GotCLoud 1.06
Image Description: From CSG at University of Michigan
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or larger -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-06-17T15:57:12Z

Terry Gliedt: /* Test the new AMI */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
# Examples to create swap file
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
dd if=/dev/zero of=$swap bs=524288 count=16384 # 8GB swap on t1.micro
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, if it is set up properly.
Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Configure s3cmd. This will ask for your AWS ID and Secret Key. If creates a file in ~/.s3cfg

<code>
s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: password_you_do_not_need_to_know
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf
Encryption password: password_you_do_not_need_to_know
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/home/ubuntu/.s3cfg'
</code>

'''(4)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

# Make sure we have a swap file
a=`swapon -s | grep -v Filename'
if [ "$a" = "" ]; then
echo "#######################################################"
echo "# You have no SWAP file set up"
echo "#"
echo "# swap=/mnt/swapfile"
echo "# sudo dd if=/dev/zero of=$swap bs=1073741824 count=20"
echo "# sudo chown root:root $swap"
echo "# sudo mkswap $swap"
echo "# sudo chmod 0600 $swap"
echo "# sudo swapon $swap"
echo "#"
echo "# If need be, add to /etc/fstab"
echo "# echo "$swap none swap sw 0 0" >> /etc/fstab"
echo "#######################################################"
fi

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name GotCLoud 1.06
Image Description: From CSG at University of Michigan
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or larger -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-06-17T15:40:17Z

Terry Gliedt: /* Set Up Swap Space */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
# Examples to create swap file
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
dd if=/dev/zero of=$swap bs=524288 count=16384 # 8GB swap on t1.micro
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, if it is set up properly.
Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Configure s3cmd. This will ask for your AWS ID and Secret Key. If creates a file in ~/.s3cfg

<code>
s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: password_you_do_not_need_to_know
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf
Encryption password: password_you_do_not_need_to_know
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/home/ubuntu/.s3cfg'
</code>

'''(4)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

# Make sure we have a swap file
a=`swapon -s | grep -v Filename'
if [ "$a" = "" ]; then
echo "#######################################################"
echo "# You have no SWAP file set up"
echo "#"
echo "# swap=/mnt/swapfile"
echo "# sudo dd if=/dev/zero of=$swap bs=1073741824 count=20"
echo "# sudo chown root:root $swap"
echo "# sudo mkswap $swap"
echo "# sudo chmod 0600 $swap"
echo "# sudo swapon $swap"
echo "#"
echo "# If need be, add to /etc/fstab"
echo "# echo "$swap none swap sw 0 0" >> /etc/fstab"
echo "#######################################################"
fi

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name GotCLoud 1.06
Image Description: From CSG at University of Michigan
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-06-17T15:28:02Z

Terry Gliedt: /* Set Up Swap Space */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
# Examples to create swap file
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
dd if=/dev/zero of=$swap bs=524288 count=30720 # 8GB swap on t1.micro
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, if it is set up properly.
Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Configure s3cmd. This will ask for your AWS ID and Secret Key. If creates a file in ~/.s3cfg

<code>
s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: password_you_do_not_need_to_know
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf
Encryption password: password_you_do_not_need_to_know
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/home/ubuntu/.s3cfg'
</code>

'''(4)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

# Make sure we have a swap file
a=`swapon -s | grep -v Filename'
if [ "$a" = "" ]; then
echo "#######################################################"
echo "# You have no SWAP file set up"
echo "#"
echo "# swap=/mnt/swapfile"
echo "# sudo dd if=/dev/zero of=$swap bs=1073741824 count=20"
echo "# sudo chown root:root $swap"
echo "# sudo mkswap $swap"
echo "# sudo chmod 0600 $swap"
echo "# sudo swapon $swap"
echo "#"
echo "# If need be, add to /etc/fstab"
echo "# echo "$swap none swap sw 0 0" >> /etc/fstab"
echo "#######################################################"
fi

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name GotCLoud 1.06
Image Description: From CSG at University of Michigan
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-06-17T13:54:24Z

Terry Gliedt: /* Configure the Host to be Usable */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, if it is set up properly.
Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Configure s3cmd. This will ask for your AWS ID and Secret Key. If creates a file in ~/.s3cfg

<code>
s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: password_you_do_not_need_to_know
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf
Encryption password: password_you_do_not_need_to_know
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/home/ubuntu/.s3cfg'
</code>

'''(4)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

# Make sure we have a swap file
a=`swapon -s | grep -v Filename'
if [ "$a" = "" ]; then
echo "#######################################################"
echo "# You have no SWAP file set up"
echo "#"
echo "# swap=/mnt/swapfile"
echo "# sudo dd if=/dev/zero of=$swap bs=1073741824 count=20"
echo "# sudo chown root:root $swap"
echo "# sudo mkswap $swap"
echo "# sudo chmod 0600 $swap"
echo "# sudo swapon $swap"
echo "#"
echo "# If need be, add to /etc/fstab"
echo "# echo "$swap none swap sw 0 0" >> /etc/fstab"
echo "#######################################################"
fi

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name GotCLoud 1.06
Image Description: From CSG at University of Michigan
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T14:30:24Z

Terry Gliedt: /* Install the Software */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, if it is set up properly.
Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Configure s3cmd. This will ask for your AWS ID and Secret Key. If creates a file in ~/.s3cfg

<code>
s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: password_you_do_not_need_to_know
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf
Encryption password: password_you_do_not_need_to_know
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/home/ubuntu/.s3cfg'
</code>

'''(4)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name GotCLoud 1.06
Image Description: From CSG at University of Michigan
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T14:19:52Z

Terry Gliedt: /* Create the AMI */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, if it is set up properly.
Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name GotCLoud 1.06
Image Description: From CSG at University of Michigan
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T14:18:30Z

Terry Gliedt: /* Install the Software */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, if it is set up properly.
Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name csg-biopipe_instance
Image Description: Image for CSG Biopipe instance
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T14:00:23Z

Terry Gliedt: /* Install the Software */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, but our recent experience is that accessing S3 data this
way (at least for 1000 Genomes data) is seldom really usable.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name csg-biopipe_instance
Image Description: Image for CSG Biopipe instance
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T13:52:03Z

Terry Gliedt: /* Configure the Host to be Usable */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, but our recent experience is that accessing S3 data this
way (at least for 1000 Genomes data) is seldom really usable.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name csg-biopipe_instance
Image Description: Image for CSG Biopipe instance
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T13:49:42Z

Terry Gliedt: /* Install the Software */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, but our recent experience is that accessing S3 data this
way (at least for 1000 Genomes data) is seldom really usable.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Follow the instructions to install the [[Pipeline Debian Package|'''GotCloud Debian packages''']]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name csg-biopipe_instance
Image Description: Image for CSG Biopipe instance
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T13:48:53Z

Terry Gliedt: /* Install the Software */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

'''(1)''' There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get update
sudo apt-get upgrade # Apply maintenance

sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''(2)''' '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, but our recent experience is that accessing S3 data this
way (at least for 1000 Genomes data) is seldom really usable.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

'''(3)''' Follow the instructions to install a Debian package [[Pipeline Debian Package|debian package]]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name csg-biopipe_instance
Image Description: Image for CSG Biopipe instance
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T13:47:44Z

Terry Gliedt: /* Make Sure System Has Swap Space */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Set Up Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

(1) There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

(2) '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, but our recent experience is that accessing S3 data this
way (at least for 1000 Genomes data) is seldom really usable.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

(3) Follow the instructions to install a Debian package [[Pipeline Debian Package|debian package]]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name csg-biopipe_instance
Image Description: Image for CSG Biopipe instance
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T13:47:23Z

Terry Gliedt: /* Install the Software */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Make Sure System Has Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Install the Software ==

(1) There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

(2) '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, but our recent experience is that accessing S3 data this
way (at least for 1000 Genomes data) is seldom really usable.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

(3) Follow the instructions to install a Debian package [[Pipeline Debian Package|debian package]]
Run the tests to be sure everything is OK.

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name csg-biopipe_instance
Image Description: Image for CSG Biopipe instance
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T13:47:06Z

Terry Gliedt: /* Make Sure System Has Swap Space */

Creating an AMI on EC2

2013-05-21T13:46:42Z

Terry Gliedt: /* Install Additional Software */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Install the Software ==

(1) There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

(2) '''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, but our recent experience is that accessing S3 data this
way (at least for 1000 Genomes data) is seldom really usable.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

(3) Follow the instructions to install a Debian package [[Pipeline Debian Package|debian package]]
Run the tests to be sure everything is OK.

== Make Sure System Has Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name csg-biopipe_instance
Image Description: Image for CSG Biopipe instance
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T13:45:38Z

Terry Gliedt: /* Install the GotCloud Software */

Creating an AMI on EC2

2013-05-21T13:45:07Z

Terry Gliedt: /* Create the AMI */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Install the GotCloud Software ==

Follow the instructions to install a Debian package [[Pipeline Debian Package|debian package]]
Run the tests to be sure everything is OK.

== Install Additional Software ==

There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, but our recent experience is that accessing S3 data this
way (at least for 1000 Genomes data) is seldom really usable.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

== Make Sure System Has Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

Once your instance is all ready with the files you want, swap space etc, then create the AMI.
In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name csg-biopipe_instance
Image Description: Image for CSG Biopipe instance
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-21T13:35:05Z

Terry Gliedt: /* Configure the Host to be Usable */

'''Notes About Creating a New EC2 AMI'''

Back to parent: [[GotCloud]]

The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process.
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

== Launch an instance ==

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-east-1'''.
Launch a new instance which we will use to set up the software and ultimately save it as an AMI.

<code>
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
</code>

== Install the GotCloud Software ==

Follow the instructions to install a Debian package [[Pipeline Debian Package|debian package]]
Run the tests to be sure everything is OK.

== Install Additional Software ==

There are a number of additional Debian packages that you may well need, so we make
sure they are all installed.

<code>
sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

'''S3fs''' allows one to access S3 storage as a conventional file system.
This can be quite handy, but our recent experience is that accessing S3 data this
way (at least for 1000 Genomes data) is seldom really usable.
Still if you're lucky, your data will be useful.
Install the software like this:

<code>
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
</code>

== Make Sure System Has Swap Space ==

Issue the command '''swapon -s''' to see if there is swap space.
If there is only a header line, you need to add a swap file like this:

<code>
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=1073741824 count=15 # 15GB swap
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab

swapon -s # Should show the swap device
</code>

== Configure the Host to be Usable ==

It is useful to configure /etc/rc.local to do most things you need at boot time.
There are many other ways to do this, but here's one simple way - create the
file /etc/rc.local (as root).
The following example sets up access details for '''s3cmd''' and '''s3fs'''
(use your own credentials).

<code>
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=passwd-s3fs
S3ERR=/tmp/s3fs.err

# Set up for GotCloud Assumes /dev/xvdf has reference files for GotCloud
mkdir -p /gotcloud
mount /dev/xvdf /gotcloud
if [ ! -d /gotcloud/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is not set up on /gotcloud"
echo "#######################################################"
fi

# Setup 1000g access by s3fs
usermod -aG fuse $USER
echo 'AKIAIW5TQEUWZ3YCZF2Q:ft1eJa3WGzNE8iitNlbA08x/G8iMqkMIkJjFCIGf' > /etc/$FILES3
chown root.root /etc/$FILES3
chmod 640 /etc/$FILES3
mkdir -p $THOUSANDG
chown $USER.$USER $THOUSANDG
# It is tempting to use caching with -o use_cache=/tmp 1000genomes
# But s3fs cache is exceedingly dumb and does not use a least recently used
# mechanism -- which will guarantee your root volume will fill up
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################" >> $S3ERR
echo "# 1000genomes is not set up on $THOUSANDG" >> $S3ERR
echo "#######################################################" >> $S3ERR
fi
df -h

exit 0
</code>

== Create the AMI ==

In your browser at the EC2 Management Console do the following:

<code>
Create Image
Image Name csg-biopipe_instance
Image Description: Image for CSG Biopipe instance
Volume Size: 30GB
Take defaults otherwise
</code>

This will take several minutes to complete.
In the EC2 Dashboard, you can monitor the progress.
When it is done, you'll see a new AMI under the list of AMIs.

Your new AMI should look pretty much like this:

<code>
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
</code>

== Test the new AMI ==

Launch a new AMI instance and check that files are in the correct places.
In the EC2 Management Console do:

<code>
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or whatever -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
</code>

Creating an AMI on EC2

2013-05-20T17:33:00Z

Terry Gliedt: /* Configure the Host to be Usable */

Creating an AMI on EC2

2013-05-20T17:28:58Z

Terry Gliedt: /* Configure the Host so it Usable */

Creating an AMI on EC2

2013-05-20T17:22:16Z

Terry Gliedt:

Creating an AMI on EC2

2013-05-20T17:19:15Z

Terry Gliedt:

Creating a Snapshot on EC2

2013-05-20T17:12:48Z

Terry Gliedt:

Back to parent: [[GotCloud]]

'''We no longer provide a snapshot for GotCLoud'''. This is no longer used.

The following are notes taken when creating a Snapshot of a volume where the GotCloud
software and data are installed.

'''Create an Instance with a Master GotCloud Volume'''

<code>
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
</code>

Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in '''us-west-2'''.

Launch a new instance which we will use to set up the software and ultimately save it as an AMI.
Your instance can use any memory size (t1.micro is fine), but should be running
'''Ubuntu Server 12.04.1 LTS 64 bit'''. The root volume size does not matter either.

Create a volume to serve as the GotCloud master volume if you do not already have one.
The volume size should be 20G at the moment.
Be sure the zone is the same as the one for the instance you just launched/created.
The volume should be based on 'No Snapshot'.
Once it is created, attach it to the instance you have launched and '''mount it it /gotcloud'''.

'''Install the Software'''

Follow the instructions to install a Debian package [[Pipeline Debian Package|debian package]]
Run the tests to be sure everything is OK.
This installs files in /usr/local/gotcloud and these need to be copied to the snapshot disk.

<code>
cd /gotcloud
rm -rf bin scripts test # Remove old GotCloud
sudo rsync -av /usr/local/gotcloud/* /gotcloud
sudo chown -R ubuntu.ubuntu bin scripts test release_version.txt
</code>

Even though the references are not part of the distribution, you cannot do anything useful
without references, so get them and make them available in /gotcloud

<code>
cd /mnt # Because the *.tar.gz will be too big for /home
sudo wget share.sph.umich.edu:gotcloud/CHANGETHIS.tar.gz
cd /gotcloud
sudo tar xzf /mnt/CHANGETHIS.tar.gz
sudo chown -R ubuntu.ubuntu gotcloud.ref
sudo chmod 0755 gotcloud.ref
chmod 0644 gotcloud.ref/*
ls -la ref
sudo rm -f /mnt/CHANGETHIS.tar.gz # if no error
cd /gotcloud/examples/umakeRef ############# still true?
ln -s /gotcloud/ref/hs37d5.fa . ############# still true?
</code>

'''Create the SnapShot'''

In your browser at the EC2 Management Console choose ''Elastic Block Storage'' -> ''Volumes''.
Find the volume for current master you created above.
Choose ''Actions'' -> ''Create Snapshot''.

<code>
Name: GotCloud Distribution
Description: GotCloud from UMICH Version 1.03a6
Note snapshot name e.g. snap-9e782bb8
</code>

Now make the name of this snapshot available to the user population.
'''In your Unix terminal''', get the snapshots.txt file, update it and make it available:

<code>
wget share.sph.umich.edu:gotcloud/snapshots.txt
vi snapshots.txt # Replace snapshot id with new one
scp snapshots.txt gotcloud@share.sph.umich.edu:
</code>

GotCloud

2013-05-20T17:10:59Z

Terry Gliedt: /* Install GotCloud Software */

= Genomes on the Cloud (GotCloud) Introduction=

To handle the increasing volume of next generation sequencing and genotyping data, we created and developed software pipelines called '''Genomes on the Cloud (GotCloud).'''

GotCloud contains Mapping & Variant Calling Pipelines.

Key Features:
* Connects sequence analysis tools together in automated pipeline
** Alignment, quality control, variant calling
* Robust against unexpected system failure using GNU make
** easy restart after failure
* Massively parallel, can run hundreds of jobs
** Splits large jobs into many pieces
** Simplifies running on clusters
* Scalable to tens of thousands of samples
* Easy to use - Automates series of configurable steps
** user doesn't have to understand/configure/know the many tools required to create high quality results
* Available on Amazon Web Services (AWS) Elastic Compute Cloud (EC2)
* Run on local machines/clusters
* Available via Debian Packages

GotCloud incorporates the alignment and variant calling pipelines that we have been using at UM into one easy to use, publicly available tool. GotCloud can run on a user's computer, on an instance in a
compute cloud, and/or can split the work up onto a cluster of machines or instances.

== Getting Help with GotCloud ==

Please join in the [http://groups.google.com/group/GotCloud GotCloud Google Group] to ask / discuss / comment about these pipelines.

Currently the "join" button appears to be missing. Click "NEW TOPIC", then select "Join this group". You can then cancel the message post (or post a message).

See [[GotCloud: FAQs]] if you have any questions. If your questions are not answered there, join the google group and ask, or email Mary Kate Wing (mktrost@umich.edu).

== Sequence Analysis Background Information ==

There are many essential steps in the analysis of next generation sequence data.

Next generation sequence data analysis starts with [http://en.wikipedia.org/wiki/FASTQ_format FASTQ files], the typical format provided from your sequencing center containing the sequence & base quality information for your data.

The fastq files are processed using the alignment pipeline which finds the most likely genomic location for each read and stores that information in a [[BAM|BAM (Binary Sequence Alignment/Map format) file]]. In addition to the sequence and base quality information contained in FASTQ files, a BAM file also contains the genomic location and some additional information about the mapping. As part of the alignment pipeline, the base qualities are adjusted to more accurately reflect the likelihood that the base is correct.

The alignment pipeline can be skipped if you already have Deduped and Recalibrated BAM files.

The variant calling pipeline processes the deduped and recalibrated BAMs file produced by the alignment pipeline or that you provide it, generating an initial list of polymorphic sites and genotypes stored in a [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) file]. The variant calling pipeline then filters the variants using both hard and a [[SVM Filtering|Support Vector Machine (SVM)]]. It then uses haplotype information to refine these genotypes in an updated VCF file.

After completing the GotCloud Variant Calling PIpeline, [[EPACTS|EPACTS (Efficient and Parallelizable Association Container Toolbox)]] can be used to perform statistical tests to identify genome-wide association from sequence data.

[[File:GotCloudDiagram.jpg|500px]]

= GotCloud Setup =

You may run the GotCloud software in several modes:
* On your own hardware running Ubuntu or Redhat/CentOS. See the instructions about installing the software below.
* On an Amazon Elastic Compute Cloud using Ubuntu or Redhat/CentOS if you do not have your own set of machines.
** See [[Amazon Snapshot]] for more information.
** You can run on an EC2 cluster instance created by StarCluster.

GotCloud has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested.

== Install GotCloud Software ==

You can install gotCloud on your system as (follow the links for the appropriate instructions):
* [[GotCloud: Source Releases|tar release]] - contains the uncompiled source
* [[GotCloud: Debian Package|debian package]] - requires root permissions to install
* As a red hat package - not yet available - requires root permissions to install

[[GotCloud: Versions]] describes the changes added to each version.

Alternatively if you are using Amazon EC2, you can use the following instructions:
* Create a machine instance based on the AMI we provide [[Creating an AMI on EC2]]
'''or''' Create an EC2 cluster instance using [[StarCluster|StarCluster]]
* Prepare your data for processing by GotCloud [[AWS Resources|AWS Resources]]

For more information on Amazon Web Services, see: https://aws.amazon.com/

== GotCloud Reference/Resource Files ==
In order to run gotCloud, you need to provide Genetic Reference and Resource Files.

These include information about the reference sequence and dbnsp positions.

See: [[GotCloud: Genetic Reference and Resource Files]] for information about the required files. It contains a description of the required files, information about generating your own versions, as well as a downloadable set of files.

== Configure GotCloud ==
* [[Configure GotCloud|Configure Gotcloud]] for your installation

== Running GotCloud Software ==

* [[GotCloud: Alignment Pipeline|Alignment Pipeline]]
* [[GotCloud: Variant Calling Pipeline|Variant Calling Pipeline]]

[[Tutorial: GotCloud]]

== Development Notes ==
* [[Creating Packages]]
* Amazon EC2
** [[Creating a Snapshot on EC2]]
** [[Creating an AMI on EC2]]
** [[Mount S3 Volume]]
** Notes on sequence data preparation in [[Amazon Storage|Amazon Storage]].

GotCloud

2013-05-20T17:10:23Z

Terry Gliedt: /* Install GotCloud Software */

= Genomes on the Cloud (GotCloud) Introduction=

To handle the increasing volume of next generation sequencing and genotyping data, we created and developed software pipelines called '''Genomes on the Cloud (GotCloud).'''

GotCloud contains Mapping & Variant Calling Pipelines.

Key Features:
* Connects sequence analysis tools together in automated pipeline
** Alignment, quality control, variant calling
* Robust against unexpected system failure using GNU make
** easy restart after failure
* Massively parallel, can run hundreds of jobs
** Splits large jobs into many pieces
** Simplifies running on clusters
* Scalable to tens of thousands of samples
* Easy to use - Automates series of configurable steps
** user doesn't have to understand/configure/know the many tools required to create high quality results
* Available on Amazon Web Services (AWS) Elastic Compute Cloud (EC2)
* Run on local machines/clusters
* Available via Debian Packages

GotCloud incorporates the alignment and variant calling pipelines that we have been using at UM into one easy to use, publicly available tool. GotCloud can run on a user's computer, on an instance in a
compute cloud, and/or can split the work up onto a cluster of machines or instances.

== Getting Help with GotCloud ==

Please join in the [http://groups.google.com/group/GotCloud GotCloud Google Group] to ask / discuss / comment about these pipelines.

Currently the "join" button appears to be missing. Click "NEW TOPIC", then select "Join this group". You can then cancel the message post (or post a message).

See [[GotCloud: FAQs]] if you have any questions. If your questions are not answered there, join the google group and ask, or email Mary Kate Wing (mktrost@umich.edu).

== Sequence Analysis Background Information ==

There are many essential steps in the analysis of next generation sequence data.

Next generation sequence data analysis starts with [http://en.wikipedia.org/wiki/FASTQ_format FASTQ files], the typical format provided from your sequencing center containing the sequence & base quality information for your data.

The fastq files are processed using the alignment pipeline which finds the most likely genomic location for each read and stores that information in a [[BAM|BAM (Binary Sequence Alignment/Map format) file]]. In addition to the sequence and base quality information contained in FASTQ files, a BAM file also contains the genomic location and some additional information about the mapping. As part of the alignment pipeline, the base qualities are adjusted to more accurately reflect the likelihood that the base is correct.

The alignment pipeline can be skipped if you already have Deduped and Recalibrated BAM files.

The variant calling pipeline processes the deduped and recalibrated BAMs file produced by the alignment pipeline or that you provide it, generating an initial list of polymorphic sites and genotypes stored in a [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) file]. The variant calling pipeline then filters the variants using both hard and a [[SVM Filtering|Support Vector Machine (SVM)]]. It then uses haplotype information to refine these genotypes in an updated VCF file.

After completing the GotCloud Variant Calling PIpeline, [[EPACTS|EPACTS (Efficient and Parallelizable Association Container Toolbox)]] can be used to perform statistical tests to identify genome-wide association from sequence data.

[[File:GotCloudDiagram.jpg|500px]]

= GotCloud Setup =

You may run the GotCloud software in several modes:
* On your own hardware running Ubuntu or Redhat/CentOS. See the instructions about installing the software below.
* On an Amazon Elastic Compute Cloud using Ubuntu or Redhat/CentOS if you do not have your own set of machines.
** See [[Amazon Snapshot]] for more information.
** You can run on an EC2 cluster instance created by StarCluster.

GotCloud has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested.

== Install GotCloud Software ==

You can install gotCloud on your system as (follow the links for the appropriate instructions):
* [[GotCloud: Source Releases|tar release]] - contains the uncompiled source
* [[GotCloud: Debian Package|debian package]] - requires root permissions to install
* As a red hat package - not yet available - requires root permissions to install

[[GotCloud: Versions]] describes the changes added to each version.

Alternatively if you are using Amazon EC2, you can use the following instructions:
* Create a machine instance based on the AMI we provide [[Creating an AMI on EC2]]
'''or''' Create an EC2 cluster instance using [[StarCluster|StarCluster]]
* [[AWS Resources|AWS Resources]]
For more information on Amazon Web Services, see: https://aws.amazon.com/

== GotCloud Reference/Resource Files ==
In order to run gotCloud, you need to provide Genetic Reference and Resource Files.

These include information about the reference sequence and dbnsp positions.

See: [[GotCloud: Genetic Reference and Resource Files]] for information about the required files. It contains a description of the required files, information about generating your own versions, as well as a downloadable set of files.

== Configure GotCloud ==
* [[Configure GotCloud|Configure Gotcloud]] for your installation

== Running GotCloud Software ==

* [[GotCloud: Alignment Pipeline|Alignment Pipeline]]
* [[GotCloud: Variant Calling Pipeline|Variant Calling Pipeline]]

[[Tutorial: GotCloud]]

== Development Notes ==
* [[Creating Packages]]
* Amazon EC2
** [[Creating a Snapshot on EC2]]
** [[Creating an AMI on EC2]]
** [[Mount S3 Volume]]
** Notes on sequence data preparation in [[Amazon Storage|Amazon Storage]].

GotCloud

2013-05-20T17:09:39Z

Terry Gliedt: /* Install GotCloud Software */

= Genomes on the Cloud (GotCloud) Introduction=

To handle the increasing volume of next generation sequencing and genotyping data, we created and developed software pipelines called '''Genomes on the Cloud (GotCloud).'''

GotCloud contains Mapping & Variant Calling Pipelines.

Key Features:
* Connects sequence analysis tools together in automated pipeline
** Alignment, quality control, variant calling
* Robust against unexpected system failure using GNU make
** easy restart after failure
* Massively parallel, can run hundreds of jobs
** Splits large jobs into many pieces
** Simplifies running on clusters
* Scalable to tens of thousands of samples
* Easy to use - Automates series of configurable steps
** user doesn't have to understand/configure/know the many tools required to create high quality results
* Available on Amazon Web Services (AWS) Elastic Compute Cloud (EC2)
* Run on local machines/clusters
* Available via Debian Packages

GotCloud incorporates the alignment and variant calling pipelines that we have been using at UM into one easy to use, publicly available tool. GotCloud can run on a user's computer, on an instance in a
compute cloud, and/or can split the work up onto a cluster of machines or instances.

== Getting Help with GotCloud ==

Please join in the [http://groups.google.com/group/GotCloud GotCloud Google Group] to ask / discuss / comment about these pipelines.

Currently the "join" button appears to be missing. Click "NEW TOPIC", then select "Join this group". You can then cancel the message post (or post a message).

See [[GotCloud: FAQs]] if you have any questions. If your questions are not answered there, join the google group and ask, or email Mary Kate Wing (mktrost@umich.edu).

== Sequence Analysis Background Information ==

There are many essential steps in the analysis of next generation sequence data.

Next generation sequence data analysis starts with [http://en.wikipedia.org/wiki/FASTQ_format FASTQ files], the typical format provided from your sequencing center containing the sequence & base quality information for your data.

The fastq files are processed using the alignment pipeline which finds the most likely genomic location for each read and stores that information in a [[BAM|BAM (Binary Sequence Alignment/Map format) file]]. In addition to the sequence and base quality information contained in FASTQ files, a BAM file also contains the genomic location and some additional information about the mapping. As part of the alignment pipeline, the base qualities are adjusted to more accurately reflect the likelihood that the base is correct.

The alignment pipeline can be skipped if you already have Deduped and Recalibrated BAM files.

The variant calling pipeline processes the deduped and recalibrated BAMs file produced by the alignment pipeline or that you provide it, generating an initial list of polymorphic sites and genotypes stored in a [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) file]. The variant calling pipeline then filters the variants using both hard and a [[SVM Filtering|Support Vector Machine (SVM)]]. It then uses haplotype information to refine these genotypes in an updated VCF file.

After completing the GotCloud Variant Calling PIpeline, [[EPACTS|EPACTS (Efficient and Parallelizable Association Container Toolbox)]] can be used to perform statistical tests to identify genome-wide association from sequence data.

[[File:GotCloudDiagram.jpg|500px]]

= GotCloud Setup =

You may run the GotCloud software in several modes:
* On your own hardware running Ubuntu or Redhat/CentOS. See the instructions about installing the software below.
* On an Amazon Elastic Compute Cloud using Ubuntu or Redhat/CentOS if you do not have your own set of machines.
** See [[Amazon Snapshot]] for more information.
** You can run on an EC2 cluster instance created by StarCluster.

GotCloud has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested.

== Install GotCloud Software ==

You can install gotCloud on your system as (follow the links for the appropriate instructions):
* [[GotCloud: Source Releases|tar release]] - contains the uncompiled source
* [[GotCloud: Debian Package|debian package]] - requires root permissions to install
* As a red hat package - not yet available - requires root permissions to install

[[GotCloud: Versions]] describes the changes added to each version.

Alternatively if you are using Amazon EC2, you can use the following instructions:
* Create a machine instance based on the AMI we provide [[Creating an AMI on EC2]]
* Create an EC2 cluster instance using [[StarCluster|StarCluster]]
* [[AWS Resources|AWS Resources]]
For more information on Amazon Web Services, see: https://aws.amazon.com/

== GotCloud Reference/Resource Files ==
In order to run gotCloud, you need to provide Genetic Reference and Resource Files.

These include information about the reference sequence and dbnsp positions.

See: [[GotCloud: Genetic Reference and Resource Files]] for information about the required files. It contains a description of the required files, information about generating your own versions, as well as a downloadable set of files.

== Configure GotCloud ==
* [[Configure GotCloud|Configure Gotcloud]] for your installation

== Running GotCloud Software ==

* [[GotCloud: Alignment Pipeline|Alignment Pipeline]]
* [[GotCloud: Variant Calling Pipeline|Variant Calling Pipeline]]

[[Tutorial: GotCloud]]

== Development Notes ==
* [[Creating Packages]]
* Amazon EC2
** [[Creating a Snapshot on EC2]]
** [[Creating an AMI on EC2]]
** [[Mount S3 Volume]]
** Notes on sequence data preparation in [[Amazon Storage|Amazon Storage]].

GotCloud: Binary Releases

2013-05-20T17:07:46Z

Terry Gliedt:

=GotCloud Debian Packages=

==Installing Debian Packages==

Back to parent: [[GotCloud]]

The GotCloud software is distributed in several Debian packages. These can be installed on traditional Debian systems as well as Ubuntu (where it has been tested). The software should run on all versions from Ubuntu Lucid (9.04) through Oneric (12.10). It has not been tested on other Debian systems, but it is likely it will work on those systems too. This software should work on any Unix system - only requiring recompilation from the source files.

The software is distributed in a single package including both the ''aligner'' and ''variant caller''. There is also a package to test the installation.

The debian files are available for download from ftp://share.sph.umich.edu/gotcloud/ :
<code>
gotcloud-bin_latest_amd64.deb aligner and variant caller
gotcloud-test_latest_amd64.deb test data for software
M.n/gotcloud-bin_M.n_amd64.deb version M.n of aligner and variant caller
M.n/gotcloud-test_M.n_amd64.deb version M.n of test data for software
</code>
Where 'M.n' is replaced with the specific version numbers.

See [[GotCloud: Versions]] for details on the changes for each version.

===Fetch the GotCloud Debian Packages===

<code>
# The easiest way:
mkdir debs
cd debs
wget --retr-symlinks ftp://share.sph.umich.edu/gotcloud/gotcloud\*latest\*.deb

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> mget gotcloud*latest*.deb
ftp> quit
221 Goodbye.
</code>

===Install the GotCloud Debian bin Package===

Installation requires one to have root access and uses dpkg which Debian users will recognize.
We recommend you install the first one separately so you can easily
identify messages about prerequisite files that need to be installed:

<code>
sudo dpkg -i ~/debs/gotcloud-bin_*_amd64.deb
Selecting previously unselected package gotcloud-bin.
(Reading database ... 25372 files and directories currently installed.)
Unpacking gotcloud-bin (from .../gotcloud-bin_1.0_amd64.deb) ...
Setting up gotcloud-bin (1.0) ...
#============================================================
# 'java' is not installed, do 'sudo apt-get install java-common default-jre'
#============================================================
#============================================================
# 'make' is not installed, do 'sudo apt-get install make'
#============================================================
#============================================================
# 'libssl0.9.8' is not installed, do 'sudo apt-get install libssl0.9.8'
#============================================================
...
</code>

If there were messages about missing packages, install them now

<code>
sudo apt-get update
sudo apt-get install java-common default-jre make libssl0.9.8
</code>

===Testing the GotCloud Installation===

We recommend that at least the first time, you install the test packages so you can
conveniently test the installation and make sure everything runs smoothly.
The tests run within a few minutes and are self checking, so unless you see obvious errors,
you can be reasonably sure everything is set up properly.
You only need to do this once, unless you have made significant changes to your Unix system.

<code>
sudo dpkg -i ~/debs/gotcloud-test*_amd64.deb
Selecting previously unselected package gotcloud-test.
(Reading database ... 27376 files and directories currently installed.)
Unpacking gotcloud-test (from .../gotcloud-test_1.0_amd64.deb) ...
Setting up gotcloud-test (1.0) ...
#======================================================================
# GotCloud README
#======================================================================
To test the GotCloud aligner, run:
gotcloud align --test ~/testalign
This will create/clear the output directory ~/testalign.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

To test the GotCloud snp caller, run:
gotcloud snpcall --test ~/testsnp
This will create/clear the output directory ~/testsnp.
Test results and a log file are put in this directory.
Results are self-checked and if errors should occur, it will be obvious.

</code>

Login as a normal user (ubuntu, not as root) and do:

<code>
# Test the aligner (2 minutes +/- 1 minute)
'''/usr/local/gotcloud/bin/gotcloud align -test ~/testalign'''
Removing any previous results from: /home/myuser/testalign/aligntest
Running GOTCLOUD TEST, test log in: /home/myuser/testalign/aligntest.log
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample2.Makefile
Created /home/myuser/testalign/aligntest/Makefiles/align_Sample1.Makefile
---------------------------------------------------------------------
Waiting while samples are processed...
Processing finished in 63 secs with no errors reported
Results from DIFF will be in /home/myuser/testalign/diff_logfiles_results.txt
Successful comparison of data in '/home/myuser/testalign/aligntest' and '/usr/local/gotcloud/test/align/expected/aligntest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testalign''' # If no error

# Test snp calling (5 minutes +/-2 minute)
'''/usr/local/gotcloud/bin/gotcloud snpcall --test ~/testsnp'''
Removing any previous results from: /home/myuser/testsnp/umaketest
Running GOTCLOUD TEST, test log in: /home/myuser/testsnp/umaketest.log
Results from DIFF will be in /home/myuser/testsnp/diff_logfiles_results_umake.txt
Successful comparison of data in '/home/myuser/testsnp/umaketest' and '/usr/local/gotcloud/test/umake/expected/umaketest'
Successfully ran the test case, congratulations!
'''rm -rf ~/testsnp''' # If no error
</code>

==Creating Your Own Debian Package From the Source==

The process to create your own Debian package from the source files is straight-forward
for Debian users. The source files actually contain source for the executables,
the test data and the scripts and configuration files used by the users.
These steps build two debian package files - one for the test data (test) and one for the
binaries and scripts (bin).

In order to simplify the building of two packages, the debian directory contains the control files for both packages (bin and test).
This requires that the normal control files (e.g. rules) be named rules.test and rules.bin
and the shell script (makedeb.sh) be aware of this nuance.
In addition the shell script can make an RPM file (using alien) and will help manage
the version for GotCloud.

'''Fetch the Source Including the Test Files'''

<code>
# The easiest way:
wget ftp://share.sph.umich.edu/gotcloud/gotcloudWithTest_latest.tgz

# Another way:
ftp share.sph.umich.edu
Connected to share.sph.umich.edu.
220 (vsFTPd 2.3.5)
Name (share.sph.umich.edu:tpg): anonymous
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> cd gotcloud
250 Directory successfully changed.
ftp> get gotcloudWithTest_latest.tgz
ftp> quit
221 Goodbye.

# Extract the source files
cd ~/src # Or wherever you want the source files
tar xzf path/gotcloudWithTest_latest.tgz
</code>

'''Fetch the Source Excluding the Test Files'''

Follow the above logic, but use: gotcloud_latest.tgz

'''Fetch just the Test Files'''

Follow the above logic, but use: gotcloudTest_latest.tgz

===Create the Debian Package Files===

<code>
'''cd ~/src''' # Or wherever you put the source files
'''cd gotcloud*'''

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh bin ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export C...
[lines deleted]
Generate package: cd gotcloud-bin; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-bin' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-bin_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
Converting symlinks into the actual executables to be distributed
[lines deleted]
dpkg-deb: building package `gotcloud-bin' in `./gotcloud-bin_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 3592780 Feb 19 11:59 gotcloud-bin_1.01_amd64.deb

# Add -replace only if rebuilding the same version
'''debian/makedeb.sh test ='''
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 ...
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
[lines deleted]
Generate package: cd gotcloud-test; dpkg-buildpackage -b -us -uc -rfakeroot
===> Version '1.01 of 'gotcloud-test' will install into 'usr/local/gotcloud'
install -d debian/tmp/DEBIAN
cp debian/control debian/postinst debian/postrm debian/tmp/DEBIAN
install -d debian/tmp/usr/share/doc/gotcloud-test_1.01
sed -e s/99.99/1.01/g < debian/changelog > debian/changelog.new
[lines deleted]
dpkg-deb: building package `gotcloud-test' in `./gotcloud-test_1.01_amd64.deb'.
Package file now in /home/myuser/dev/gotcloud
-rw-r--r-- 1 myuser myuser 433269556 Feb 19 13:49 gotcloud-test_1.01_amd64.deb
</code>

The deb files will be created in the directory above the debian directory (e.g. gotcloud)

Amazon Snapshot

2013-05-20T17:06:39Z

Terry Gliedt:

Back to the beginning: [[GotCloud]]

'''We no longer use a snapshot.''' It is very likely that you will need quite a few packages installed
so that you can compile your software, access the EC2 application data or access data on S3.
It just seemed foolish to not make these software available in an AMI.

GotCloud is made available in various forms.
It is distributed as conventional packages for Ubuntu and as compress TAR files for others.
In addition the source is available from github.
In Amazon Web Services the software is made available as an Amazon Machine Instance (AMI).

The GotCloud software itself only requires a few packages to be installed for Ubuntu installations
(java-common default-jre make libssl0.9.8).
However, there are a number of things you may well want to do in getting your data
ready for processing (access data on S3 storage, compile GotCloud or others, or
access the EC2 application data).
Assuming this is the case, the GotCloud AMI has installed these packages on Ubunutu.
If you need to run on some other distribution, you may need to install their packages.

<code>
sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

You will almost certainly need to fetch and install your own reference files - regardless
of the details of the system you are using.
Finally, you'll need access to your FASTQ files - either copied to the Amazon instance
or perhaps accessible from S3 storage.

If the GotCloud instance is unacceptable for some reason, you may install the
software and reference files wherever you'd like
(read about this in [[Pipeline_Debian_Package|Installing from a Debian package]]).

Your first task is get an AWS account and keys so that you can use the AWS EC2 Console Dashboard
(see https://console.aws.amazon.com/ec2/).
From here you can launch instances prepared by others or create your own.
We cannot assist in this step - Amazon has plenty of documentation.
Once you are at the AWS EC2 Console Dashboard, you're almost ready for GotCloud.

'''Your First Instance'''

You'll need to know some details when launching an instance:

* '''Launch an Instance''' - use the GotCloud instance running 64 bit software.

* '''Instance size''' (memory and number of processors). The pipeline software will require at least 4GB of memory (''type m1.medium'') and can use as many processors as is available.

* '''Storage''' for the instance refers to the size for root (/) partition. This can be quite small, as little as 8GB can work. Of course if you intend to bring other files/programs to the instance, you may need to increase this to something a bit larger (e.g. 30GB).

* '''Data Storage''' for the aligner or SNP caller (see below)

'''Prepare Your Instance'''

You will also want additional storage volumes for:

* '''Local Storage''' for the instance refers to the size for root (/) partition. This can be quite small, as little as 8GB can work. Of course if you intend to bring other files/programs to the instance, you may need to increase this to something a bit larger (e.g. 30GB).

* '''Data Storage''' for the aligner or SNP caller will likely be far larger than the system you are creating.
You'll need to create EBS Volumes for the input and output of the aligner and SNP caller.

'''Prepare Your Storage'''

These can be quite substantial and because of that we recommend you create separate volumes like this:

* Your '''input FASTQ''' files for the aligner.
This may have been done for you by some vendor when they put your FASTQ data on an S3 volume.
If so, your vendor will need to provide you with the details of how to access your FASTQ files.
If your FASTQ files are not in S3 storage, you'll have to create a volume for this and copy your data into it.
This can take a very long time.

* The '''output of the aligner''' (BAM files)

* The '''intermediate files of the SNP caller''' (GLF files)

* The '''final output of the SNP caller''' (VCF files)

Amazon Snapshot

2013-05-20T17:03:57Z

Terry Gliedt:

Amazon Snapshot

2013-05-20T16:58:28Z

Terry Gliedt:

Back to the beginning: [[GotCloud]]

GotCloud is made available in various forms.
It is distributed as conventional packages for Ubuntu and as compress TAR files for others.
In addition the source is available from github.
In Amazon Web Services the software is made available as an Amazon Machine Instance (AMI).

The GotCloud software itself only requires a few packages to be installed for Ubuntu installations
(java-common default-jre make libssl0.9.8).
However, there are a number of things you may well want to do in getting your data
ready for processing (access data on S3 storage, compile GotCloud or others, or
access the EC2 application data).
Assuming this is the case, the GotCloud AMI has installed these packages on Ubunutu.
If you need to run on some other distribution, you may need to install their packages.

<code>
sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
</code>

You will almost certainly need to fetch and install your own reference files - regardless
of the details of the system you are using.
Finally, you'll need access to your FASTQ files - either copied to the Amazon instance
or perhaps accessible from S3 storage.

If the GotCloud instance is unacceptable for some reason, you may install the
software and reference files wherever you'd like
(read about this in [[Pipeline_Debian_Package|Installing from a Debian package]]).

Your first task is get an AWS account and keys so that you can use the AWS EC2 Console Dashboard
(see https://console.aws.amazon.com/ec2/).
From here you can launch instances prepared by others or create your own.
We cannot assist in this step - Amazon has plenty of documentation.
Once you are at the AWS EC2 Console Dashboard, you're almost ready for GotCloud.

'''Your First Instance'''

You'll need to know some details when launching an instance:

* '''Launch an Instance''' - use the GotCloud instance running 64 bit software.

* '''Instance size''' (memory and number of processors). The pipeline software will require at least 4GB of memory (''type m1.medium'') and can use as many processors as is available.

* '''Storage''' for the instance refers to the size for root (/) partition. This can be quite small, as little as 8GB can work. Of course if you intend to bring other files/programs to the instance, you may need to increase this to something a bit larger (e.g. 30GB).

* '''Data Storage''' for the aligner or SNP caller (see below)

'''Prepare Your Instance'''

You will also want additional storage volumes for:

* '''Local Storage''' for the instance refers to the size for root (/) partition. This can be quite small, as little as 8GB can work. Of course if you intend to bring other files/programs to the instance, you may need to increase this to something a bit larger (e.g. 30GB).

* '''Data Storage''' for the aligner or SNP caller will likely be far larger than the system you are creating.
You'll need to create EBS Volumes for the input and output of the aligner and SNP caller.
These can be quite substantial and because of that we recommend you create separate volumes like this:

* Your '''input FASTQ''' files for the aligner. This may have been done for you by some vendor when they put your FASTQ data on an S3 volume. If so, your vendor will need to provide you with the details of how to access your FASTQ files.

* The '''output of the aligner''' (BAM files)

* The '''intermediate files of the SNP caller''' (GLF files)

* The '''final output of the SNP caller''' (VCF files)

<code>
wget -qO - share.sph.umich.edu:gotcloud/snapshots.txt

# GotCloud SnapShot List
#
# Create an EBS volume from these snapshots. Use the AWS console or
# with an ec2-api-tools command:
#
# ec2-create-volume -K ~/ec2/EC2-X509-private_key.pem \
# -C ~/ec2/EC2-X509-cert.pem -s 40 \
# --snapshot snap-14ea7632 --region us-west-2 -z us-west-2a
#
# Availability
# Zone Snapshot Size
us-west-2a snap-14ea7632 40GB
</code>

This will create a device which you need to mount in your instance.
This will create a device like /dev/sdf, which unfortunately actually translates to
the device /dev/xvdf in your Linux instance. Once the volume is ready, mount it
by logging into your instance with ssh and issuing the command:

<code>
sudo mkdir -p /gotcloud
sudo mount /dev/xvdf /gotcloud # or whatever device yours is
df -h
</code>

This will make the GotCloud software available under the path /gotcloud/bin etc.
Each time your instance is started, you'll need to mount this volume.
You may want to create a small shell script to mount the device.

In '''Your Data''' the storage volumes will vary based on what you data you have.
The sequence data might already exist, provided by a vendor who created the sequence data.
If not, you'll have to create a volume for this and copy your data into it.
You'll have to mount volumes for all three types of data (sequence, aligner and umake).

You should expect the three data volumes will all need to be the same size. That is, if your sequence data is 300GB, then you'll need an additional 300GB for the aligner output and then another 300GB of storage for the umake output. We suggest you consider making each set of data be separate volumes.

You may also find that your sequence data is too large to be easily handled in one go,
so you might choose to only use the aligner/umake on part of your sequence data, capture the files
of interest from umake, and then go back and rerun the software with the next bit of sequence data.