Changes

From Genome Analysis Wiki
Jump to navigationJump to search

GotCloud (view source)

Revision as of 18:09, 15 December 2015

2,611 bytes added ,  18:09, 15 December 2015
→‎GotCloud Setup
Line 1: Line 1: −
= Genomes on the Cloud (GotCloud) Introduction=
+
__TOC__
 +
 
 +
== Genomes on the Cloud (GotCloud) Introduction==
    
To handle the increasing volume of next generation sequencing and genotyping data, we created and developed software pipelines called '''Genomes on the Cloud (GotCloud).'''
 
To handle the increasing volume of next generation sequencing and genotyping data, we created and developed software pipelines called '''Genomes on the Cloud (GotCloud).'''
Line 18: Line 20:  
* Available on Amazon Web Services (AWS) Elastic Compute Cloud (EC2)
 
* Available on Amazon Web Services (AWS) Elastic Compute Cloud (EC2)
 
* Run on local machines/clusters
 
* Run on local machines/clusters
−
* Available via Debian Packages
      
GotCloud incorporates the alignment and variant calling pipelines that we have been using at UM into one easy to use, publicly available tool.  GotCloud can run on a user's computer, on an instance in a  
 
GotCloud incorporates the alignment and variant calling pipelines that we have been using at UM into one easy to use, publicly available tool.  GotCloud can run on a user's computer, on an instance in a  
 
compute cloud, and/or can split the work up onto a cluster of machines or instances.  
 
compute cloud, and/or can split the work up onto a cluster of machines or instances.  
 +
 +
[[File:Gotcloud.puzzles.v2.png|500px]]
      −
== Getting Help with GotCloud ==
+
=== Getting Help with GotCloud ===
    
Please join in the [http://groups.google.com/group/GotCloud GotCloud Google Group] to ask / discuss / comment about these pipelines.
 
Please join in the [http://groups.google.com/group/GotCloud GotCloud Google Group] to ask / discuss / comment about these pipelines.
Line 33: Line 36:       −
== Sequence Analysis Background Information ==
+
=== Sequence Analysis Background Information ===
    
There are many essential steps in the analysis of next generation sequence data.  
 
There are many essential steps in the analysis of next generation sequence data.  
Line 39: Line 42:  
Next generation sequence data analysis starts with [http://en.wikipedia.org/wiki/FASTQ_format FASTQ files], the typical format provided from your sequencing center containing the sequence & base quality information for your data.
 
Next generation sequence data analysis starts with [http://en.wikipedia.org/wiki/FASTQ_format FASTQ files], the typical format provided from your sequencing center containing the sequence & base quality information for your data.
   −
The fastq files are processed using the alignment pipeline which finds the most likely genomic location for each read and stores that information in a [[BAM|BAM (Binary Sequence Alignment/Map format) file]].  In addition to the sequence and base quality information contained in FASTQ files, a BAM file also contains the genomic location and some additional information about the mapping.  As part of the alignment pipeline, the base qualities are adjusted to more accurately reflect the likelihood that the base is correct.  
+
The fastq files are processed using the [[GotCloud: Alignment Pipeline|alignment pipeline]] which finds the most likely genomic location for each read and stores that information in a [[BAM|BAM (Binary Sequence Alignment/Map format) file]].  In addition to the sequence and base quality information contained in FASTQ files, a BAM file also contains the genomic location and some additional information about the mapping.  As part of the [[GotCloud: Alignment Pipeline|alignment pipeline]], the base qualities are adjusted to more accurately reflect the likelihood that the base is correct.  
   −
The alignment pipeline can be skipped if you already have Deduped and Recalibrated BAM files.
+
The [[GotCloud: Alignment Pipeline|alignment pipeline]] can be skipped if you already have Deduped and Recalibrated BAM files.  If you have BAMs, but they needed to be deduped and recalibrated, you can use our [[GotCloud:_Alignment_Sub-Pipelines#recabQC_2|recabQC pipeline]].
   −
The variant calling pipeline processes the deduped and recalibrated BAMs file produced by the alignment pipeline or that you provide it, generating an initial list of polymorphic sites and genotypes stored in a [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) file].  The variant calling pipeline then filters the  variants using both hard filters and a [[SVM Filtering|Support Vector Machine (SVM)]].  It then uses haplotype information to refine these genotypes in an updated VCF file.
+
The [[GotCloud: Variant Calling Pipeline|variant calling pipeline]] processes the deduped and recalibrated BAM files produced by the alignment pipeline or that you provide it, generating an initial list of polymorphic sites and genotypes stored in a [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) file].  The [[GotCloud: Variant Calling Pipeline|variant calling pipeline]] then filters the  variants using both hard filters and a [[SVM Filtering|Support Vector Machine (SVM)]].  It then uses haplotype information to refine these genotypes in an updated VCF file.
    
After completing the GotCloud Variant Calling Pipeline, [[EPACTS|EPACTS (Efficient and Parallelizable Association Container Toolbox)]] can be used to perform statistical tests to identify genome-wide association from sequence data.
 
After completing the GotCloud Variant Calling Pipeline, [[EPACTS|EPACTS (Efficient and Parallelizable Association Container Toolbox)]] can be used to perform statistical tests to identify genome-wide association from sequence data.
Line 49: Line 52:  
[[File:GotCloudDiagram.jpg|500px]]
 
[[File:GotCloudDiagram.jpg|500px]]
   −
= GotCloud Setup =
+
 
 +
== Publication ==
 +
If you use GotCloud, please cite our publication:
 +
[http://genome.cshlp.org/content/early/2015/04/14/gr.176552.114.abstract Jun, Goo, et al. "An efficient and scalable analysis framework for variant extraction and refinement from population scale DNA sequence data." Genome research (2015): gr-176552.]
 +
 
 +
== GotCloud Setup ==
    
You may run the GotCloud software in several modes:
 
You may run the GotCloud software in several modes:
 
* On your own hardware running Ubuntu or Redhat/CentOS. See the instructions about installing the software below.
 
* On your own hardware running Ubuntu or Redhat/CentOS. See the instructions about installing the software below.
 
* On an Amazon Elastic Compute Cloud using Ubuntu or Redhat/CentOS if you do not have your own set of machines.
 
* On an Amazon Elastic Compute Cloud using Ubuntu or Redhat/CentOS if you do not have your own set of machines.
−
** See [[Amazon Snapshot]] for more information.
+
** See [[GotCloud: Amazon]] for more information.
 
** You can run on an EC2 cluster instance created by StarCluster.  
 
** You can run on an EC2 cluster instance created by StarCluster.  
   −
GotCloud has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS.  While it should work on other Linux systems, they have not yet been tested.  
+
GotCloud has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS and Red Hat 6.6.  While it should work on other Linux systems, they have not yet been tested.  
 +
 
 +
=== GotCloud on Amazon ===
 +
You can take advantage of GotCloud AMI's when running on Amazon.  The GotCloud already includes GotCloud and default reference files.
 +
 
 +
See [[GotCloud: Amazon]] for instructions on using GotCloud on Amazon.
 +
 
 +
=== GotCloud Setup on Any Linux Machine ===
 +
 
 +
==== GotCloud Dependencies ====
 +
 
 +
GotCloud requires certain things to be installed in order to run:
 +
* perl - gotcloud is a perl script and it calls many other perl scripts
 +
** Zlib.pm - required for perl scripts to read compressed files.
 +
* make - GNU make is used to run the pipelines
 +
* java - required to run the beagle step of the ld-aware genotype refinement
 +
* curses/ncurses (required for samtools)
 +
** On Ubuntu: <code>sudo apt-get install libncurses5 libncurses5-dev</code>
 +
* cmake (required for premo)
 +
** On Ubuntu: <code>sudo apt-get install cmake</code>
 +
 
 +
You can check if your system has the proper software installed by invoking the command
 +
[gotcloud_path]/scripts/check_requirements.sh
   −
== Install GotCloud Software ==
+
==== Install GotCloud Software ====
    
You can install gotCloud on your system as (follow the links for the appropriate instructions):
 
You can install gotCloud on your system as (follow the links for the appropriate instructions):
−
* [[GotCloud: Source Releases|tar release]] - contains the uncompiled source
+
* [[GotCloud: Source Releases|source release]] - contains the scripts and uncompiled source
−
* [[GotCloud: Debian Package|debian package]] - requires root permissions to install
+
* [[GotCloud: Binary Releases|binary release]] - contains the scripts and pre-compiled binaries (no source)
−
* As a red hat package - not yet available - requires root permissions to install
      
[[GotCloud: Versions]] describes the changes added to each version.
 
[[GotCloud: Versions]] describes the changes added to each version.
   −
Alternatively if you are using Amazon EC2, you can use the following instructions:
+
 
−
* Create a machine instance based on the AMI we provide [[Creating an AMI on EC2]]
+
Alternatively if you are using Amazon EC2, you can use one of the following sets of instructions:
−
'''or''' Create an EC2 cluster instance using [[StarCluster|StarCluster]]
+
* Create a machine instance based on the AMI we provide: [[Amazon Single Node]]
−
* Prepare your data for processing by GotCloud [[AWS Resources|AWS Resources]]
+
* Create an EC2 cluster instance using [[StarCluster|StarCluster]]
 +
For more information on using GotCloud on Amazon, see: [[GotCloud: Amazon]]
    
For more information on Amazon Web Services, see: https://aws.amazon.com/
 
For more information on Amazon Web Services, see: https://aws.amazon.com/
   −
== GotCloud Reference/Resource Files ==
+
==== GotCloud Reference/Resource Files ====
 
In order to run gotCloud, you need to provide Genetic Reference and Resource Files.
 
In order to run gotCloud, you need to provide Genetic Reference and Resource Files.
   Line 81: Line 111:     
See: [[GotCloud: Genetic Reference and Resource Files]] for information about the required files.  It contains a description of the required files, information about generating your own versions, as well as a downloadable set of files.
 
See: [[GotCloud: Genetic Reference and Resource Files]] for information about the required files.  It contains a description of the required files, information about generating your own versions, as well as a downloadable set of files.
 +
* When running on Amazon, a default set of reference files are included in the GotCloud AMI.
   −
 
+
==== Configure GotCloud ====
−
== Configure GotCloud ==
   
* [[Configure GotCloud|Configure Gotcloud]] for your installation
 
* [[Configure GotCloud|Configure Gotcloud]] for your installation
   Line 89: Line 119:     
* [[GotCloud: Alignment Pipeline|Alignment Pipeline]]  
 
* [[GotCloud: Alignment Pipeline|Alignment Pipeline]]  
 +
** [[GotCloud: Alignment Sub-Pipelines|Alignment Sub-Pipelines]] - for if you do not want to run the entire Alignment Pipeline
 
* [[GotCloud: Variant Calling Pipeline|Variant Calling Pipeline]]
 
* [[GotCloud: Variant Calling Pipeline|Variant Calling Pipeline]]
 +
* Indel Calling Pipeline
 +
* [[GotCloud: GenomeSTRiP Pipeline|GenomeSTRiP Pipeline]] (Structural Variation)
 +
* MEI Calling Pipeline - ''Ask if you're interested''
 +
 +
You can also create your own pipelines.  Instructions are here:
 +
* [[GotCloud: Creating a New Pipeline]]
 +
 +
=== GotCloud Demos ===
 +
GotCloud Demos (originally from our sequencing workshop):
 +
* [[SeqShop: Sequence Mapping and Assembly Practical]]
 +
* [[SeqShop: Variant Calling and Filtering for SNPs Practical]]
 +
* [[SeqShop: Variant Calling and Filtering for INDELs Practical]]
 +
* [[SeqShop: Analysis of Structural Variation Practical]]
 +
 +
GotCloud on Amazon Demo (snpcall & indel):
 +
* [[GotCloud: Amazon Demo]]
   −
[[Tutorial: GotCloud]]
+
Deprecated: [[Tutorial: GotCloud]]
   −
== Development Notes ==
+
== UMich Development/Release How-To Notes ==
−
* [[Creating Packages]]
+
* [[Releasing GotCloud]]
 
* Amazon EC2
 
* Amazon EC2
−
** [[Creating a Snapshot on EC2]]
   
** [[Creating an AMI on EC2]]
 
** [[Creating an AMI on EC2]]
 +
** [[Creating a Snapshot on EC2]] (deprecated)
 
** [[Mount S3 Volume]]
 
** [[Mount S3 Volume]]
 
** Notes on sequence data preparation in [[Amazon Storage|Amazon Storage]].
 
** Notes on sequence data preparation in [[Amazon Storage|Amazon Storage]].
 +
 +
* [[Git_FAQs#Subtrees|Upgrade Git Subtree]]
Pjvh
61

edits

Retrieved from "http://genome.sph.umich.edu/wiki/Special:MobileDiff/7713...13964"

Navigation menu