Amazon Snapshot - Revision history

Terry Gliedt at 17:06, 20 May 2013

2013-05-20T17:06:39Z

← Older revision		Revision as of 13:06, 20 May 2013
Line 1:		Line 1:
	Back to the beginning: [[GotCloud]]		Back to the beginning: [[GotCloud]]

			'''We no longer use a snapshot.''' It is very likely that you will need quite a few packages installed
			so that you can compile your software, access the EC2 application data or access data on S3.
			It just seemed foolish to not make these software available in an AMI.



	GotCloud is made available in various forms.		GotCloud is made available in various forms.

Terry Gliedt at 17:03, 20 May 2013

2013-05-20T17:03:57Z

@@ Line 57: / Line 57: @@
 * '''Data Storage''' for the aligner or SNP caller will likely be far larger than the system you are creating.
 You'll need to create EBS Volumes for the input and output of the aligner and SNP caller.
 These can be quite substantial and because of that we recommend you create separate volumes like this:
-* Your '''input FASTQ''' files for the aligner. This may have been done for you by some vendor when they put your FASTQ data on an S3 volume. If so, your vendor will need to provide you with the details of how to access your FASTQ files.
+* Your '''input FASTQ''' files for the aligner.
 * The '''output of the aligner''' (BAM files)
@@ Line 66: / Line 73: @@
 * The '''final output of the SNP caller''' (VCF files)

Terry Gliedt at 16:58, 20 May 2013

2013-05-20T16:58:28Z

← Older revision		Revision as of 12:58, 20 May 2013
Line 59:		Line 59:
	These can be quite substantial and because of that we recommend you create separate volumes like this:		These can be quite substantial and because of that we recommend you create separate volumes like this:

	* Your input FASTQ files for the aligner. This may have been done for you by some vendor when they put your FASTQ data on an S3 volume. If so, your vendor will need to provide you with the details of how to access your FASTQ files.		* Your '''input FASTQ''' files for the aligner. This may have been done for you by some vendor when they put your FASTQ data on an S3 volume. If so, your vendor will need to provide you with the details of how to access your FASTQ files.

	* The output of the aligner (BAM files)		* The '''output of the aligner''' (BAM files)

	* The intermediate files of the SNP caller (GLF files)		* The '''intermediate files of the SNP caller''' (GLF files)

	* The final output of the SNP caller (VCF files)		* The '''final output of the SNP caller''' (VCF files)

	<code>		<code>

Terry Gliedt at 16:57, 20 May 2013

2013-05-20T16:57:36Z

← Older revision		Revision as of 12:57, 20 May 2013
Line 44:		Line 44:
	* '''Instance size''' (memory and number of processors). The pipeline software will require at least 4GB of memory (''type m1.medium'') and can use as many processors as is available.		* '''Instance size''' (memory and number of processors). The pipeline software will require at least 4GB of memory (''type m1.medium'') and can use as many processors as is available.

	* '''Storage''' for the instance refers to the size for root (/) partition. This can be quite small, as little as 8GB can work. Of course if you intend to bring ~~lots of~~ other files/programs to the instance, you may ~~want~~ to increase this to something a bit larger (e.g. 30GB).		* '''Storage''' for the instance refers to the size for root (/) partition. This can be quite small, as little as 8GB can work. Of course if you intend to bring other files/programs to the instance, you may need to increase this to something a bit larger (e.g. 30GB).

	* '''Data Storage''' for the aligner or ~~snpcaller will likely be far larger than the system you are creating.~~		* '''Data Storage''' for the aligner or SNP caller (see below)
	~~You'll need to create EBS Volumes for the input and output of the aligner and snpcaller.~~
	~~These can be quite substantial and because of that we recommend you create separate volumes like this:~~

	* Your input FASTQ files for the aligner. This might have been done for you by some vendor when they put your FASTQ data on an S3 volume. If so, your vendor will need to provide you with the details of how to access your FASTQ files.

	* The output of the aligner (BAM files)		'''Prepare Your Instance'''

	* The intermediate files of the SNP caller		You will also want additional storage volumes for:

			* '''Local Storage''' for the instance refers to the size for root (/) partition. This can be quite small, as little as 8GB can work. Of course if you intend to bring other files/programs to the instance, you may need to increase this to something a bit larger (e.g. 30GB).

	'''~~Prepare Your Instance~~'''		* '''Data Storage''' for the aligner or SNP caller will likely be far larger than the system you are creating.
			You'll need to create EBS Volumes for the input and output of the aligner and SNP caller.
			These can be quite substantial and because of that we recommend you create separate volumes like this:

	~~You~~ will ~~also want additional storage volumes for:~~		* Your input FASTQ files for the aligner. This may have been done for you by some vendor when they put your FASTQ data on an S3 volume. If so, your vendor will need to provide you with the details of how to access your FASTQ files.

	* ~~GotCloud software and reference files~~		* The output of the aligner (BAM files)
	* Your data
	** Sequence data
	** Output of the aligner
	** Output of umake

	The ~~'''first of these''' is a small volume based on a snapshot containing the GotCloud~~ files ~~you will need.~~		* The intermediate files of the SNP caller (GLF files)
	~~We provide an AWS snapshot~~ of ~~a small volume which contains~~ the ~~aligner and umake software and reference~~ files.
	~~Create an EBS volume based on our snapshot and then mount that volume on your instance.~~
	~~In the EC2 Management Console under ELASTIC BLOCK STORE, select Volumes -> Create Volume.~~
	~~In the prompt supply the size and Snapshot (based on the table below~~).
	~~You may take the defaults for the Volume Type and IOPS.~~

	The ~~snapshot ID varies by zone and the release~~ of the ~~software. You can see the complete list of GotCloud snapshots:~~		* The final output of the SNP caller (VCF files)

	<code>		<code>

Terry Gliedt at 16:53, 20 May 2013

2013-05-20T16:53:36Z

← Older revision		Revision as of 12:53, 20 May 2013
Line 2:		Line 2:

	GotCloud is made available in various forms.		GotCloud is made available in various forms.
	It is distributed as conventional packages for Ubuntu and ~~Redhat/CentOS systems~~.		It is distributed as conventional packages for Ubuntu and as compress TAR files for others.
	In Amazon Web Services the software is made available ~~in a EBS~~ (~~Elastic Block Store~~) ~~'''Snapshot'''.~~		In addition the source is available from github.
	~~This is simple a copy of a data volume we have created that has our software~~		In Amazon Web Services the software is made available as an Amazon Machine Instance (AMI).
	~~and, additionally, some reference files you will find useful.~~
	~~You need to create your own EBS volume from our snapshot, mount your~~
	~~new volume and you are ready~~.

	If ~~this does not work~~ or is unacceptable for some reason, you may install the		The GotCloud software itself only requires a few packages to be installed for Ubuntu installations
			(java-common default-jre make libssl0.9.8).
			However, there are a number of things you may well want to do in getting your data
			ready for processing (access data on S3 storage, compile GotCloud or others, or
			access the EC2 application data).
			Assuming this is the case, the GotCloud AMI has installed these packages on Ubunutu.
			If you need to run on some other distribution, you may need to install their packages.

			<code>
			sudo apt-get install java-common default-jre make libssl0.9.8
			sudo apt-get install libnet-amazon-ec2-perl
			sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
			</code>

			You will almost certainly need to fetch and install your own reference files - regardless
			of the details of the system you are using.
			Finally, you'll need access to your FASTQ files - either copied to the Amazon instance
			or perhaps accessible from S3 storage.

			If the GotCloud instance is unacceptable for some reason, you may install the
	software and reference files wherever you'd like		software and reference files wherever you'd like
	(read about this in [[Pipeline_Debian_Package\|Installing from a Debian package]]).		(read about this in [[Pipeline_Debian_Package\|Installing from a Debian package]]).
Line 24:		Line 40:
	You'll need to know some details when launching an instance:		You'll need to know some details when launching an instance:

	* '''Launch an Instance''' - use ~~any~~ instance running 64 bit software ~~and either an Ubuntu of any version or Redhat/CentOS 6.3 distribution~~.		* '''Launch an Instance''' - use the GotCloud instance running 64 bit software.

	* '''Instance size''' (memory and number of processors). The pipeline software will require at least 4GB of memory (''type m1.medium'') and can use as many processors as is available.		* '''Instance size''' (memory and number of processors). The pipeline software will require at least 4GB of memory (''type m1.medium'') and can use as many processors as is available.

	* '''~~GotCloud Volume~~''' (~~copy from GotCloud~~ ''~~snapshot~~''). ~~We provide an AWS snapshot~~ of ~~a small volume which contains~~ the aligner and ~~umake software~~ and ~~reference~~ files. ~~Your task is to create~~ an ~~EBS~~ volume ~~based on our snapshot and then mount that volume on~~ your ~~instance~~ (~~see below for more precise details~~).		* '''Storage''' for the instance refers to the size for root (/) partition. This can be quite small, as little as 8GB can work. Of course if you intend to bring lots of other files/programs to the instance, you may want to increase this to something a bit larger (e.g. 30GB).

			* '''Data Storage''' for the aligner or snpcaller will likely be far larger than the system you are creating.
			You'll need to create EBS Volumes for the input and output of the aligner and snpcaller.
			These can be quite substantial and because of that we recommend you create separate volumes like this:

			* Your input FASTQ files for the aligner. This might have been done for you by some vendor when they put your FASTQ data on an S3 volume. If so, your vendor will need to provide you with the details of how to access your FASTQ files.

			* The output of the aligner (BAM files)

	* ~~'''Storage''' for the instance refers to the size for root (/) partition. This can be quite small, as little as 8GB should work. Of course if you intend to bring lots~~ of ~~other files/programs to~~ the ~~instance, you may want to increase this to something a bit larger (e.g. 30GB).~~		* The intermediate files of the SNP caller

Terry Gliedt at 19:47, 19 November 2012

2012-11-19T19:47:07Z

← Older revision		Revision as of 15:47, 19 November 2012
Line 53:		Line 53:

	<code>		<code>
	wget -qO - share.sph.umich.edu:gotcloud/snapshots.txt		wget -qO - share.sph.umich.edu:gotcloud/snapshots.txt

	# GotCloud SnapShot List		# GotCloud SnapShot List
	#		#
	# Create an EBS volume from these snapshots. Use the AWS console or		# Create an EBS volume from these snapshots. Use the AWS console or
	# with an ec2-api-tools command:		# with an ec2-api-tools command:
	#		#
	# ec2-create-volume -K ~/ec2/EC2-X509-private_key.pem \		# ec2-create-volume -K ~/ec2/EC2-X509-private_key.pem \
	# -C ~/ec2/EC2-X509-cert.pem -s 40 \		# -C ~/ec2/EC2-X509-cert.pem -s 40 \
	# --snapshot snap-14ea7632 --region us-west-2 -z us-west-2a		# --snapshot snap-14ea7632 --region us-west-2 -z us-west-2a
	#		#
	# Availability		# Availability
	# Zone Snapshot Size		# Zone Snapshot Size
	us-west-2a snap-14ea7632 40GB		us-west-2a snap-14ea7632 40GB
	</code>		</code>
	~~Choose the correct snapshot-id and create the volume.~~
	~~Do not format the volume. Do not mount it.~~
	~~Rather you will need to tell StarCluster to mount this volume on each node in the cluster.~~
	~~This is done ny~~


	This will create a device which you need to mount in your instance.		This will create a device which you need to mount in your instance.

Terry Gliedt at 19:40, 19 November 2012

2012-11-19T19:40:16Z

← Older revision		Revision as of 15:40, 19 November 2012
Line 49:		Line 49:
	In the prompt supply the size and Snapshot (based on the table below).		In the prompt supply the size and Snapshot (based on the table below).
	You may take the defaults for the Volume Type and IOPS.		You may take the defaults for the Volume Type and IOPS.

			The snapshot ID varies by zone and the release of the software. You can see the complete list of GotCloud snapshots:

	<code>		<code>
	Availability		wget -qO - share.sph.umich.edu:gotcloud/snapshots.txt
	~~Name~~ Zone Snapshot Size
	~~GotCloud distribution~~ us-west-2a snap-14ea7632 40GB		# GotCloud SnapShot List
			#
			# Create an EBS volume from these snapshots. Use the AWS console or
			# with an ec2-api-tools command:
			#
			# ec2-create-volume -K ~/ec2/EC2-X509-private_key.pem \
			# -C ~/ec2/EC2-X509-cert.pem -s 40 \
			# --snapshot snap-14ea7632 --region us-west-2 -z us-west-2a
			#
			# Availability
			# Zone Snapshot Size
			us-west-2a snap-14ea7632 40GB
	</code>		</code>
			Choose the correct snapshot-id and create the volume.
			Do not format the volume. Do not mount it.
			Rather you will need to tell StarCluster to mount this volume on each node in the cluster.
			This is done ny


	~~and create the volume.~~ This will create a device which you need to mount in your instance.		This will create a device which you need to mount in your instance.
	This will create a device like /dev/sdf, which unfortunately actually translates to		This will create a device like /dev/sdf, which unfortunately actually translates to
	the device /dev/xvdf in your Linux instance. Once the volume is ready, mount it		the device /dev/xvdf in your Linux instance. Once the volume is ready, mount it

Terry Gliedt at 18:52, 19 November 2012

2012-11-19T18:52:03Z

← Older revision		Revision as of 14:52, 19 November 2012
Line 1:		Line 1:
	Back to the beginning, [[GotCloud]]		Back to the beginning: [[GotCloud]]

	GotCloud is made available in various forms.		GotCloud is made available in various forms.

Terry Gliedt at 18:51, 19 November 2012

2012-11-19T18:51:37Z

← Older revision		Revision as of 14:51, 19 November 2012
Line 1:		Line 1:
	Back to the beginning [[GotCloud]]		Back to the beginning, [[GotCloud]]

	GotCloud is made available in various forms.		GotCloud is made available in various forms.

Terry Gliedt at 18:51, 19 November 2012

2012-11-19T18:51:26Z

← Older revision		Revision as of 14:51, 19 November 2012
Line 1:		Line 1:
	Back to the beginning [GotCloud~~\|\|http://genome.sph.umich.edu/wiki/GotCloud~~]		Back to the beginning [[GotCloud]]

	GotCloud is made available in various forms.		GotCloud is made available in various forms.