Changes

From Genome Analysis Wiki
Jump to navigationJump to search

GotCloud: GenomeSTRiP Pipeline (view source)

Revision as of 18:47, 10 February 2015

8,305 bytes added ,  18:47, 10 February 2015
→‎Configuration File
Line 40: Line 40:     
== Input Data ==
 
== Input Data ==
 +
== Input Data==
 +
* [[#BAM Files|Aligned/Processed/Recalibrated BAM files]]
 +
* [[#BAM List File|BAM list file containing Sample IDs & BAM file names]]
 +
* [[#Reference Files|Reference files]]
 +
* [[#Configuration File|Configuration file to override default options]]
 +
 +
=== BAM Files ===
 +
The BAM files need to be duplicate-marked and base-quality recalibrated in order to obtain high quality SNP calls. Generating these BAM files from original FASTQs is automatically done as part of the [[Alignment Pipeline]] of GotCloud.
 +
 +
=== BAM List File ===
 +
* Automatically created when running the GotCloud [[Alignment Pipeline]]
 +
* Each line of the BAM list file represents a single individual
 +
 +
Columns:
 +
# sample id
 +
# comma separated population labels (optional column)
 +
# BAM File 1 (preferable to have full paths to BAM files)
 +
# BAM File 2 (if more than 1 BAM per sample)
 +
:...
 +
 +
: # BAM File N (if more than 1 BAM per sample)
 +
[SAMPLE_ID]    [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...
 +
or
 +
[SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ...
 +
 +
* Notes:
 +
** tab delimited
 +
** multiple BAMs per individual may be provided, but should all be on the same line of the list file
 +
** population label is optional - it will default to <code>ALL</code>
 +
*** only used by Thunder (part of ldrefine pipeline)
 +
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.
 +
 +
The path to the BAM List file is defaulted to the outputDirectory/bam.list. It can be overridden by setting --bamlist, --bam_list, or --list on the command-line or by setting BAM_LIST in your configuration file to the path to the BAM List File. See [[#Required Options|Required Options]] for more information.
 +
 +
=== Reference Files ===
 +
See [[GotCloud: Genetic Reference and Resource Files]] for detailed information about the multiple required reference files for the variant calling pipeline, including:
 +
* How to obtain default references
 +
* Configuration keys & default values
 +
* How to generate your own references
 +
* How to point GotCloud to your reference files
 +
 +
Required Reference File Types:
 +
* [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]]
 +
 
=== Configuration File ===
 
=== Configuration File ===
−
{{:GotCloud: Configuration}}
+
The GotCloud configuration file contains the run-time options, including software binaries and command line arguments.  A default configuration file is automatically loaded.  Users may specify their own configuration file specifying just the values different than the defaults.  The configuration file is not required if there are no values to override.
 +
 
 +
* Default GotCloud configuration file is <code>gotcloud/bin/gotcloudDefaults.conf</code>
 +
* Comments begin with a <code>#</code>
 +
* Format: <code>KEY = value </code>
 +
** where <code>KEY</code> is the item being set and <code>value</code> is its new value
 +
* Some settings can be defined both in the configuration file and on the GotCloud command-line
 +
** command-line options take priority over configuration file settings
 +
* A KEY can be used in another KEY's value by specifying $(KEY)
 +
** Example:
 +
**: <code>KEY1 = value1</code>
 +
**: <code>KEY2 = $(KEY1)/value2</code>
 +
*** When <code>KEY2</code> is used, it will be equal to: <code>value1/value2</code>
    +
====Output Directory====
 +
* The output directory is required for running GotCloud, so GotCloud knows where to write its output
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! Configuration Key !! Command-line Flag !!  colspan="3"|Value Description
 +
|-
 +
|OUT_DIR||--outdir || output directory
 +
|}
 +
 +
====Reference/Resource Files====
 +
* See [[GotCloud: Genetic Reference and Resource Files]] for reference/resource file configuration settings
 +
 +
====Cluster Configuration====
 +
Currently GotCloud/GenomeSTRiP only supports mosix and will wait for each job to complete after launching it.
 +
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! Configuration Key !! Command-line Flag !!  colspan="3"|Value Description
 +
|-
 +
| GENOMESTRIP_MOSIX_OPT || --mosix-opt || colspan="3"| options to pass to mosix, example:
 +
<pre>-j36,37,38,39,40,41,45,46,47,48,49</pre>
 +
|}
 
==== GenomeSTRiP specific configuration settings ====
 
==== GenomeSTRiP specific configuration settings ====
 
When using GenomeSTRiP, you need to specify the following configuration settings:
 
When using GenomeSTRiP, you need to specify the following configuration settings:
Line 49: Line 125:     
'''Replace the specified paths to the path to these files.'''
 
'''Replace the specified paths to the path to these files.'''
 +
 +
==GotCloud/GenomeSTRiP Command-line Options/Configuration Settings==
 +
 +
=== Required Options ===
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --outdir ''path'' || OUT_DIR || output directory ||
 +
|-
 +
| --list ''file'' || BAM_LIST || path to the [[GotCloud: Variant Calling Pipeline#BAM List File|BAM List File]] || $(OUT_DIR)/bam.list
 +
|-
 +
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
 +
|}
 +
 +
=== Common Options ===
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! colspan="4" | Common Options
 +
|-
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --conf ''file'' || || configuration file to use ||
 +
|}
 +
 +
=== Cluster Options ===
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --mosix-opt ''opts'' || GENOMESTRIP_MOSIX_OPT || options to pass to the mosix command ||
 +
|-
 +
|}
 +
 +
=== Test/Debug Options ===
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
|-
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --verbose ||  || Add additional messages when reading configuration ||
 +
|}
 +
 +
===Reference/Resource Files===
 +
* See [[GotCloud: Genetic Reference and Resource Files]] for reference/resource file configuration settings
 +
 +
=== Analysis Region Options ===
 +
See [[#Targeted/Exome Sequencing Settings|Targeted/Exome Sequencing Settings]] for more information on specifying exome/targetted regions and other settings.
 +
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --chrs ''# #'' || CHRS || space separated list of chromosomes to process || 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 X
 +
|-
 +
| --region ''#:#-#'' || GENOMESTRIP_REGION || call region - skip regions of chromosome outside of specified region
 +
format (-end is optional): chr:start-end
 +
|
 +
|-
 +
|}
 +
 +
=== Path Options ===
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
|-
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --bam-prefix ''prefix'' || BAM_PREFIX || path to prepend to relative BAM file paths in the BAM list ||
 +
|-
 +
| --ref-prefix ''prefix'' || REF_PREFIX || path to prepend to relative reference/resource file paths ||
 +
|-
 +
| --base-prefix ''prefix'' || BASE_PREFIX || path to prepend to relative paths for the BAM list file, BAM (if BAM_PREFIX isn't specified), reference/resource files (if REF_PREFIX isn't specified) ||
 +
|-
 +
| --gotcloudroot ''path'' || GOTCLOUD_ROOT || specify to use a different directory for finding GotCloud bins/scripts || based on the location of the gotcloud/umake.pl script
 +
|-
 +
|}
       
== Running GotCloud/GenomeSTRiP ==
 
== Running GotCloud/GenomeSTRiP ==
 +
The general command-line for running GenomeSTRiP via GotCloud is:
 +
gotcloud genomestrip --run-<step> --conf <gotcloud.conf> --outdir <outputDirectory> --numjobs <#>
 +
Where:
 +
* <code>--run-<step></code> indicates which pipeline to run.  Options are:
 +
** <code>--run-metadata</code> - [[#Metadata Pipeline|Metadata Pipeline]]
 +
** <code>--run-discovery</code> - [[#Discovery Pipeline|Discovery Pipeline]]
 +
** <code>--run-genotype</code> - [[#Genotyping Pipeline|Genotyping Pipeline]]
 +
** <code>--run-thirdparty</code> - [[#3rd-party Site Genotyping/Filtering Pipeline|3rd-party Site Genotyping/Filtering Pipeline]]
 +
* <code>--conf <gotcloud.conf></code> - points to the configuration file to use
 +
* <code>--outdir <outputDirectory></code> - tells GotCloud where to write the output
 +
* <code>--numjobs <#></code> - number of jobs to run in parallel
 +
 +
Optional Parameters:
 +
* <code>--metadata <metadataDirectory></code> - points to a directory containing pre-made metadata files
 +
** Only required if skipping the <code>--run-metadata</code> step.
    
=== Metadata Pipeline ===
 
=== Metadata Pipeline ===
Line 60: Line 221:  
NOTE: You don't always have to create the metadata on your own. You can in principle use the public metadata generated for 1000G samples, under the assumption that the metadata share similar characteristics to your samples. But if you have enough computing resources, the best practice is to create metadata specifically for your sequence data.
 
NOTE: You don't always have to create the metadata on your own. You can in principle use the public metadata generated for 1000G samples, under the assumption that the metadata share similar characteristics to your samples. But if you have enough computing resources, the best practice is to create metadata specifically for your sequence data.
   −
Command-line to run the metadata step:
+
Timing:
−
gotcloud genomestrip --run-metadata --conf gotcloud.conf --outdir outputDirectory --numjobs 10
+
* 10 BAMs, chr 21 and 22: 1 hour, 20 mins with 10 jobs
−
 
  −
The metadata pipeline may take a long time to run:
  −
* Chromosomes 21 & 22 for 10 BAMs took 1 hour to run with 10 jobs
   
* May take a few weeks to run whole genome on 2,000 BAMs
 
* May take a few weeks to run whole genome on 2,000 BAMs
   Line 70: Line 228:  
=== Discovery Pipeline ===
 
=== Discovery Pipeline ===
 
The discovery pipeline performs variant discovery across all samples as well as variant filtering based on expert knowledge.
 
The discovery pipeline performs variant discovery across all samples as well as variant filtering based on expert knowledge.
−
  −
gotcloud genomestrip --run-discovery --conf gotcloud.conf --outdir outputDirectory --numjobs 10
      
Timing:
 
Timing:
−
* 10 BAMs, chr 21 and 22: 1 hour, 20 mins
+
* 10 BAMs, chr 21 and 22: 1 hour, 20 mins with 10 jobs
    
=== Genotyping Pipeline ===
 
=== Genotyping Pipeline ===
 
The genotyping pipeline iterates the discovered variants across the samples, calculating the genotype likelihood for each possible genotype.
 
The genotyping pipeline iterates the discovered variants across the samples, calculating the genotype likelihood for each possible genotype.
   −
gotcloud genomestrip --run-genotype --conf gotcloud.conf --outdir outputDirectory --numjobs 10
+
Timing:
 +
* 10 BAMs, chr 21 and 22: 4 mins with 10 jobs
    
=== 3rd-party Site Genotyping/Filtering Pipeline ====
 
=== 3rd-party Site Genotyping/Filtering Pipeline ====
Retrieved from "http://genome.sph.umich.edu/wiki/Special:MobileDiff/12804...12814"

Navigation menu