Amazon Snapshot

Back to the beginning: GotCloud

GotCloud is made available in various forms. It is distributed as conventional packages for Ubuntu and as compress TAR files for others. In addition the source is available from github. In Amazon Web Services the software is made available as an Amazon Machine Instance (AMI).

The GotCloud software itself only requires a few packages to be installed for Ubuntu installations (java-common default-jre make libssl0.9.8). However, there are a number of things you may well want to do in getting your data ready for processing (access data on S3 storage, compile GotCloud or others, or access the EC2 application data). Assuming this is the case, the GotCloud AMI has installed these packages on Ubunutu. If you need to run on some other distribution, you may need to install their packages.

 sudo apt-get install java-common default-jre make libssl0.9.8 
 sudo apt-get install libnet-amazon-ec2-perl
 sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev

You will almost certainly need to fetch and install your own reference files - regardless of the details of the system you are using. Finally, you'll need access to your FASTQ files - either copied to the Amazon instance or perhaps accessible from S3 storage.

If the GotCloud instance is unacceptable for some reason, you may install the software and reference files wherever you'd like (read about this in Installing from a Debian package).

Your first task is get an AWS account and keys so that you can use the AWS EC2 Console Dashboard (see https://console.aws.amazon.com/ec2/). From here you can launch instances prepared by others or create your own. We cannot assist in this step - Amazon has plenty of documentation. Once you are at the AWS EC2 Console Dashboard, you're almost ready for GotCloud.

Your First Instance

You'll need to know some details when launching an instance:

• Launch an Instance - use the GotCloud instance running 64 bit software.

• Instance size (memory and number of processors). The pipeline software will require at least 4GB of memory (type m1.medium) and can use as many processors as is available.

• Storage for the instance refers to the size for root (/) partition. This can be quite small, as little as 8GB can work. Of course if you intend to bring lots of other files/programs to the instance, you may want to increase this to something a bit larger (e.g. 30GB).

• Data Storage for the aligner or snpcaller will likely be far larger than the system you are creating.

You'll need to create EBS Volumes for the input and output of the aligner and snpcaller. These can be quite substantial and because of that we recommend you create separate volumes like this:

• Your input FASTQ files for the aligner. This might have been done for you by some vendor when they put your FASTQ data on an S3 volume. If so, your vendor will need to provide you with the details of how to access your FASTQ files.

• The output of the aligner (BAM files)

• The intermediate files of the SNP caller

Prepare Your Instance

You will also want additional storage volumes for:

• GotCloud software and reference files
• Your data
  • Sequence data
  • Output of the aligner
  • Output of umake

The first of these is a small volume based on a snapshot containing the GotCloud files you will need. We provide an AWS snapshot of a small volume which contains the aligner and umake software and reference files. Create an EBS volume based on our snapshot and then mount that volume on your instance. In the EC2 Management Console under ELASTIC BLOCK STORE, select Volumes -> Create Volume. In the prompt supply the size and Snapshot (based on the table below). You may take the defaults for the Volume Type and IOPS.

The snapshot ID varies by zone and the release of the software. You can see the complete list of GotCloud snapshots:

 wget -qO -  share.sph.umich.edu:gotcloud/snapshots.txt

 #                           GotCloud SnapShot List
 #
 #   Create an EBS volume from these snapshots. Use the AWS console or
 #   with an ec2-api-tools command:
 #
 #     ec2-create-volume -K ~/ec2/EC2-X509-private_key.pem \
 #       -C ~/ec2/EC2-X509-cert.pem -s 40 \
 #       --snapshot snap-14ea7632 --region us-west-2 -z us-west-2a
 #
 #                         Availability
 #  Zone         Snapshot       Size
 us-west-2a     snap-14ea7632   40GB

This will create a device which you need to mount in your instance. This will create a device like /dev/sdf, which unfortunately actually translates to the device /dev/xvdf in your Linux instance. Once the volume is ready, mount it by logging into your instance with ssh and issuing the command:

 sudo mkdir -p /gotcloud
 sudo mount /dev/xvdf  /gotcloud    # or whatever device yours is
 df -h

This will make the GotCloud software available under the path /gotcloud/bin etc. Each time your instance is started, you'll need to mount this volume. You may want to create a small shell script to mount the device.

In Your Data the storage volumes will vary based on what you data you have. The sequence data might already exist, provided by a vendor who created the sequence data. If not, you'll have to create a volume for this and copy your data into it. You'll have to mount volumes for all three types of data (sequence, aligner and umake).

You should expect the three data volumes will all need to be the same size. That is, if your sequence data is 300GB, then you'll need an additional 300GB for the aligner output and then another 300GB of storage for the umake output. We suggest you consider making each set of data be separate volumes.

You may also find that your sequence data is too large to be easily handled in one go, so you might choose to only use the aligner/umake on part of your sequence data, capture the files of interest from umake, and then go back and rerun the software with the next bit of sequence data.