Amazon Storage

Back to parent [1]

Setting up your storage is perhaps the most difficult step as it is controlled completely by the size of your data. As a general rule you will need three times the space required for your sequence data. For instance in the 1000 Genomes data, the data for one individual takes about 45G. If you have 1000 Genome data for nine individuals, you'll need about 1500GB of space (9x450x3 plus a little extra space).

Making your data available for the Pipeline software can be accomplished in many ways. Here is a simple straightforward organization you might want to use.

• Launch your instance and login as explained in the AWS documentation.
• Using the AWS EC2 Console Dashboard create one EBS volume (ELASTIC BLOCK STORE -> Volumes) for the sequence data (e.g. 500GB).
• Using the Dashboard create another EBS volume for the output of the aligner step (e.g. another 500GB).
• Using the Dashboard create another EBS volume for the output of the umake step (e.g. another 500GB).

Attach the volume to the instance you have just launched and specify as a separate device: f, g and h (e.g. /dev/sdf, /dev/sdg and /dev/sdh). It'll take a few minutes for the volume to show up in your instance. Note: as of this writing if you specify a device as sdf, it will actually show up as /dev/xvdf in the instance.

This first time you need to prepare the disks by formatting and mounting them:

 sudo fdisk -l /dev/xdvg          # Do not continue until this works
   Disk /dev/xvdg: 536.9 GB, 536870912000 bytes
   255 heads, 63 sectors/track, 65270 cylinders, total 1048576000 sectors
   Units = sectors of 1 * 512 = 512 bytes
   Sector size (logical/physical): 512 bytes / 512 bytes
   I/O size (minimum/optimal): 512 bytes / 512 bytes
   Disk identifier: 0x00000000

   Disk /dev/xvdg doesn't contain a valid partition table

 sudo mkfs -t ext4 /dev/xvdg
   mke2fs 1.42 (29-Nov-2011)
   Filesystem label=
   OS type: Linux
   Block size=4096 (log=2)
   Fragment size=4096 (log=2)
   Stride=0 blocks, Stripe width=0 blocks
   32768000 inodes, 131072000 blocks
   6553600 blocks (5.00%) reserved for the super user
   First data block=0
   Maximum filesystem blocks=4294967296
   4000 block groups
   32768 blocks per group, 32768 fragments per group
   8192 inodes per group
   Superblock backups stored on blocks: 
       32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
       4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
       102400000
   Allocating group tables: done                            
   Writing inode tables: done                            
   Creating journal (32768 blocks): done
   Writing superblocks and filesystem accounting information: done     

 sudo fdisk -l /dev/xdvg          # Do not continue until this works
 sudo mkfs -t ext4 /dev/xdvg