Difference between revisions of "Amazon Storage"
Terry Gliedt (talk | contribs) |
Terry Gliedt (talk | contribs) |
||
Line 16: | Line 16: | ||
Attach the volume to the instance you have just launched and specify as a | Attach the volume to the instance you have just launched and specify as a | ||
separate device: f, g and h (e.g. /dev/sdf, /dev/sdg and /dev/sdh). | separate device: f, g and h (e.g. /dev/sdf, /dev/sdg and /dev/sdh). | ||
− | Note: as of this writing you | + | It'll take a few minutes for the volume to show up in your instance. |
+ | Note: as of this writing if you specify a device as sdf, it will actually show up as /dev/xvdf in the instance. | ||
This first time you need to prepare the disks by formatting and mounting them: | This first time you need to prepare the disks by formatting and mounting them: | ||
<code> | <code> | ||
− | sudo fdisk -l /dev/xdvg | + | '''sudo fdisk -l /dev/xdvg''' # Do not continue until this works |
+ | Disk /dev/xvdg: 536.9 GB, 536870912000 bytes | ||
+ | 255 heads, 63 sectors/track, 65270 cylinders, total 1048576000 sectors | ||
+ | Units = sectors of 1 * 512 = 512 bytes | ||
+ | Sector size (logical/physical): 512 bytes / 512 bytes | ||
+ | I/O size (minimum/optimal): 512 bytes / 512 bytes | ||
+ | Disk identifier: 0x00000000 | ||
+ | |||
+ | Disk /dev/xvdg doesn't contain a valid partition table | ||
+ | |||
+ | '''sudo mkfs -t ext4 /dev/xvdg''' | ||
+ | mke2fs 1.42 (29-Nov-2011) | ||
+ | Filesystem label= | ||
+ | OS type: Linux | ||
+ | Block size=4096 (log=2) | ||
+ | Fragment size=4096 (log=2) | ||
+ | Stride=0 blocks, Stripe width=0 blocks | ||
+ | 32768000 inodes, 131072000 blocks | ||
+ | 6553600 blocks (5.00%) reserved for the super user | ||
+ | First data block=0 | ||
+ | Maximum filesystem blocks=4294967296 | ||
+ | 4000 block groups | ||
+ | 32768 blocks per group, 32768 fragments per group | ||
+ | 8192 inodes per group | ||
+ | Superblock backups stored on blocks: | ||
+ | 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, | ||
+ | 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, | ||
+ | 102400000 | ||
+ | Allocating group tables: done | ||
+ | Writing inode tables: done | ||
+ | Creating journal (32768 blocks): done | ||
+ | Writing superblocks and filesystem accounting information: done | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | sudo fdisk -l /dev/xdvg # Do not continue until this works | ||
+ | sudo mkfs -t ext4 /dev/xdvg | ||
+ | </code> |
Revision as of 09:11, 29 October 2012
Back to parent [1]
Setting up your storage is perhaps the most difficult step as it is controlled completely by the size of your data. As a general rule you will need three times the space required for your sequence data. For instance in the 1000 Genomes data, the data for one individual takes about 45G. If you have 1000 Genome data for nine individuals, you'll need about 1500GB of space (9x450x3 plus a little extra space).
Making your data available for the Pipeline software can be accomplished in many ways. Here is a simple straightforward organization you might want to use.
- Launch your instance and login as explained in the AWS documentation.
- Using the AWS EC2 Console Dashboard create one EBS volume (ELASTIC BLOCK STORE -> Volumes) for the sequence data (e.g. 500GB).
- Using the Dashboard create another EBS volume for the output of the aligner step (e.g. another 500GB).
- Using the Dashboard create another EBS volume for the output of the umake step (e.g. another 500GB).
Attach the volume to the instance you have just launched and specify as a separate device: f, g and h (e.g. /dev/sdf, /dev/sdg and /dev/sdh). It'll take a few minutes for the volume to show up in your instance. Note: as of this writing if you specify a device as sdf, it will actually show up as /dev/xvdf in the instance.
This first time you need to prepare the disks by formatting and mounting them:
sudo fdisk -l /dev/xdvg # Do not continue until this works
Disk /dev/xvdg: 536.9 GB, 536870912000 bytes
255 heads, 63 sectors/track, 65270 cylinders, total 1048576000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/xvdg doesn't contain a valid partition table
sudo mkfs -t ext4 /dev/xvdg
mke2fs 1.42 (29-Nov-2011)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
32768000 inodes, 131072000 blocks
6553600 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
4000 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
sudo fdisk -l /dev/xdvg # Do not continue until this works
sudo mkfs -t ext4 /dev/xdvg