Amazon Storage - Revision history

Mktrost at 20:49, 5 November 2014

2014-11-05T20:49:37Z

← Older revision		Revision as of 16:49, 5 November 2014
Line 9:		Line 9:
	Here is a simple straightforward organization you might want to use.		Here is a simple straightforward organization you might want to use.

			===Making Use of Instance Storage===
			Some instances provide storage. By default they are not added to your instance. You need to set them up prior to launching your instance.

	'''~~Create Volumes~~'''		I found instructions at: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#InstanceStore_UsageScenarios
			* referred me to: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html#Using_OverridingAMIBDM

			'''Make sure you add the instance store prior to launching'''
			* After launching, only one of the 2 instance stores was mounted at /mnt
			** To mount the other instance store, I followed the instructions in the '''To make a volume available''' section of: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-add-volume-to-instance.html


			===Create Volumes===

	* Launch your instance and login as explained in the AWS documentation.		* Launch your instance and login as explained in the AWS documentation.
Line 29:		Line 39:


	~~'''~~Prepare and Attach Volumes~~'''~~		====Prepare and Attach Volumes====

	This first time you need to prepare the disks by formatting and mounting them.		This first time you need to prepare the disks by formatting and mounting them.
Line 80:		Line 90:


	~~'''~~Getting Your Sequence Data~~'''~~		===Getting Your Sequence Data===

	Now it's time to get your sequence data so you can run the Pipeline on it.		Now it's time to get your sequence data so you can run the Pipeline on it.

Amkwong at 16:01, 29 November 2012

2012-11-29T16:01:02Z

← Older revision		Revision as of 12:01, 29 November 2012
Line 4:		Line 4:
	As a general rule you will need three times the space required for your sequence data.		As a general rule you will need three times the space required for your sequence data.
	For instance in the 1000 Genomes data, the data for one individual takes about 45G.		For instance in the 1000 Genomes data, the data for one individual takes about 45G.
	If you have 1000 Genome data for nine individuals, you'll need about 1500GB of space (~~9x450x3~~ plus a little extra space).		If you have 1000 Genome data for nine individuals, you'll need about 1500GB of space (9x45x3 plus a little extra space).

	Making your data available for the Pipeline software can be accomplished in many ways.		Making your data available for the Pipeline software can be accomplished in many ways.

Mktrost at 17:43, 15 November 2012

2012-11-15T17:43:57Z

← Older revision		Revision as of 13:43, 15 November 2012
Line 1:		Line 1:
	Back to the beginning at [~~http://genome.sph.umich.edu/wiki/Pipelines~~]		Back to the beginning at [[GotCloud]]

	Setting up your storage is perhaps the most difficult step as it is controlled completely by the size of your data.		Setting up your storage is perhaps the most difficult step as it is controlled completely by the size of your data.

Terry Gliedt at 14:11, 30 October 2012

2012-10-30T14:11:00Z

← Older revision		Revision as of 10:11, 30 October 2012
Line 115:		Line 115:
	By now you know that getting your sequence data can take a long time.		By now you know that getting your sequence data can take a long time.
	When this completes you are finally ready to run the Pipeline software.		When this completes you are finally ready to run the Pipeline software.

			In one case we copied 387GB of data from 1000 Genomes to our own EBS volume.
			This took about 60 hours (longer actually because the copy failed and had to be restarted).
			For an m1.medium instance (any sized instance can be used for this step) this cost about $20 (Oct 2012).
			The cost for the 500GB EBS volume where the 1000 Genomes data was copied is very low ($0.50/month).

Terry Gliedt at 14:05, 29 October 2012

2012-10-29T14:05:56Z

← Older revision		Revision as of 10:05, 29 October 2012
Line 108:		Line 108:
	Files created		Files created
	Completed bucket 'seq/data/HG01112' in 378.38 min		Completed bucket 'seq/data/HG01112' in 378.38 min

	[lines deleted]		[lines deleted]
	</code>		</code>

Terry Gliedt at 14:05, 29 October 2012

2012-10-29T14:05:28Z

← Older revision		Revision as of 10:05, 29 October 2012
Line 39:		Line 39:
	<code>		<code>
	'''sudo fdisk -l /dev/xdvf''' # Do not continue until this works		'''sudo fdisk -l /dev/xdvf''' # Do not continue until this works
	Disk /dev/xvdf: 536.9 GB, 536870912000 bytes		Disk /dev/xvdf: 536.9 GB, 536870912000 bytes
	[lines deleted]		[lines deleted]
	Disk /dev/xvdf doesn't contain a valid partition table # This is OK		Disk /dev/xvdf doesn't contain a valid partition table # This is OK

	# Device exists, good. Format it, destroying any data there, so be sure of the device name and label.		# Device exists, good. Format it, destroying any data there, so be sure of the device name and label.
	'''sudo mkfs -t ext4 -L seq /dev/xvdf'''		'''sudo mkfs -t ext4 -L seq /dev/xvdf'''
	mke2fs 1.42 (29-Nov-2011)		mke2fs 1.42 (29-Nov-2011)
	Filesystem label=seq		Filesystem label=seq
	[lines deleted]		[lines deleted]
	Allocating group tables: done		Allocating group tables: done
	Writing inode tables: done		Writing inode tables: done
	Creating journal (32768 blocks): done		Creating journal (32768 blocks): done
	Writing superblocks and filesystem accounting information: done		Writing superblocks and filesystem accounting information: done

	# Repeat these steps for the other volumes		# Repeat these steps for the other volumes
	'''sudo fdisk -l /dev/xdvg'''		'''sudo fdisk -l /dev/xdvg'''
	'''sudo mkfs -t ext4 -L aligner /dev/xvdg'''		'''sudo mkfs -t ext4 -L aligner /dev/xvdg'''

Terry Gliedt at 14:04, 29 October 2012

2012-10-29T14:04:34Z

← Older revision		Revision as of 10:04, 29 October 2012
Line 39:		Line 39:
	<code>		<code>
	'''sudo fdisk -l /dev/xdvf''' # Do not continue until this works		'''sudo fdisk -l /dev/xdvf''' # Do not continue until this works
	Disk /dev/xvdf: 536.9 GB, 536870912000 bytes		Disk /dev/xvdf: 536.9 GB, 536870912000 bytes
	[lines deleted]		[lines deleted]
	Disk /dev/xvdf doesn't contain a valid partition table # This is OK		Disk /dev/xvdf doesn't contain a valid partition table # This is OK

	# Device exists, good. Format it, destroying any data there, so be sure of the device name and label.		# Device exists, good. Format it, destroying any data there, so be sure of the device name and label.
	'''sudo mkfs -t ext4 -L seq /dev/xvdf'''		'''sudo mkfs -t ext4 -L seq /dev/xvdf'''
	mke2fs 1.42 (29-Nov-2011)		mke2fs 1.42 (29-Nov-2011)
	Filesystem label=seq		Filesystem label=seq
	[lines deleted]		[lines deleted]
	Allocating group tables: done		Allocating group tables: done
	Writing inode tables: done		Writing inode tables: done
	Creating journal (32768 blocks): done		Creating journal (32768 blocks): done
	Writing superblocks and filesystem accounting information: done		Writing superblocks and filesystem accounting information: done

	# Repeat these steps for the other volumes		# Repeat these steps for the other volumes

Terry Gliedt at 14:03, 29 October 2012

2012-10-29T14:03:49Z

← Older revision		Revision as of 10:03, 29 October 2012
Line 43:		Line 43:
	Disk /dev/xvdf doesn't contain a valid partition table # This is OK		Disk /dev/xvdf doesn't contain a valid partition table # This is OK

	# Device exists, good. Format it, destroying any data there, so be sure of the device name.		# Device exists, good. Format it, destroying any data there, so be sure of the device name and label.
	'''sudo mkfs -t ext4 -L seq /dev/xvdf'''		'''sudo mkfs -t ext4 -L seq /dev/xvdf'''
	mke2fs 1.42 (29-Nov-2011)		mke2fs 1.42 (29-Nov-2011)

Terry Gliedt at 14:02, 29 October 2012

2012-10-29T14:02:40Z

@@ Line 8: / Line 8: @@
 Making your data available for the Pipeline software can be accomplished in many ways.
 Here is a simple straightforward organization you might want to use.
 '''Create Volumes'''
@@ Line 23: / Line 24: @@
 We suggest you create storage volumes and use each one for each particular kind of data:
-* /dev/xvdf for sequence data
+* /dev/xvdf for sequence data      This might have already been done for you
 * /dev/xvdg for aligner output data
 * /dev/xvdh for umake output data
 '''Prepare and Attach Volumes'''
@@ Line 31: / Line 33: @@
 This first time you need to prepare the disks by formatting and mounting them.
 Realize this step destroys the data on each volume, so be careful which volume you are working on.
 <code>
@@ Line 73: / Line 78: @@
    /dev/xvdh       500G  7.5G  467G   2% /home/ubuntu/myseq/umake
 </code>

Terry Gliedt at 13:36, 29 October 2012

2012-10-29T13:36:17Z

← Older revision		Revision as of 09:36, 29 October 2012
Line 1:		Line 1:
	Back to [~~[Beginning\|~~Pipelines]]		Back to the beginning at [http://genome.sph.umich.edu/wiki/Pipelines]

	Setting up your storage is perhaps the most difficult step as it is controlled completely by the size of your data.		Setting up your storage is perhaps the most difficult step as it is controlled completely by the size of your data.