Difference between revisions of "GotCloud: GenomeSTRiP Pipeline"

From Genome Analysis Wiki
Jump to navigationJump to search
 
(2 intermediate revisions by the same user not shown)
Line 85: Line 85:
  
 
=== Configuration File ===
 
=== Configuration File ===
−
{{:GotCloud: Configuration}}
+
The GotCloud configuration file contains the run-time options, including software binaries and command line arguments.  A default configuration file is automatically loaded.  Users may specify their own configuration file specifying just the values different than the defaults.  The configuration file is not required if there are no values to override.
  
 +
* Default GotCloud configuration file is <code>gotcloud/bin/gotcloudDefaults.conf</code>
 +
* Comments begin with a <code>#</code>
 +
* Format: <code>KEY = value </code>
 +
** where <code>KEY</code> is the item being set and <code>value</code> is its new value
 +
* Some settings can be defined both in the configuration file and on the GotCloud command-line
 +
** command-line options take priority over configuration file settings
 +
* A KEY can be used in another KEY's value by specifying $(KEY)
 +
** Example:
 +
**: <code>KEY1 = value1</code>
 +
**: <code>KEY2 = $(KEY1)/value2</code>
 +
*** When <code>KEY2</code> is used, it will be equal to: <code>value1/value2</code>
 +
 +
====Output Directory====
 +
* The output directory is required for running GotCloud, so GotCloud knows where to write its output
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! Configuration Key !! Command-line Flag !!  colspan="3"|Value Description
 +
|-
 +
|OUT_DIR||--outdir || output directory
 +
|}
 +
 +
====Reference/Resource Files====
 +
* See [[GotCloud: Genetic Reference and Resource Files]] for reference/resource file configuration settings
 +
 +
====Cluster Configuration====
 +
Currently GotCloud/GenomeSTRiP only supports mosix and will wait for each job to complete after launching it.
 +
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! Configuration Key !! Command-line Flag !!  colspan="3"|Value Description
 +
|-
 +
| GENOMESTRIP_MOSIX_OPT || --mosix-opt || colspan="3"| options to pass to mosix, example:
 +
<pre>-j36,37,38,39,40,41,45,46,47,48,49</pre>
 +
|}
 
==== GenomeSTRiP specific configuration settings ====
 
==== GenomeSTRiP specific configuration settings ====
 
When using GenomeSTRiP, you need to specify the following configuration settings:
 
When using GenomeSTRiP, you need to specify the following configuration settings:
Line 93: Line 125:
  
 
'''Replace the specified paths to the path to these files.'''
 
'''Replace the specified paths to the path to these files.'''
 +
 +
==GotCloud/GenomeSTRiP Command-line Options/Configuration Settings==
 +
 +
=== Required Options ===
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --outdir ''path'' || OUT_DIR || output directory ||
 +
|-
 +
| --list ''file'' || BAM_LIST || path to the [[GotCloud: Variant Calling Pipeline#BAM List File|BAM List File]] || $(OUT_DIR)/bam.list
 +
|-
 +
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
 +
|}
 +
 +
=== Common Options ===
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! colspan="4" | Common Options
 +
|-
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --conf ''file'' || || configuration file to use ||
 +
|}
 +
 +
=== Cluster Options ===
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --mosix-opt ''opts'' || GENOMESTRIP_MOSIX_OPT || options to pass to the mosix command ||
 +
|-
 +
|}
 +
 +
=== Test/Debug Options ===
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
|-
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --verbose ||  || Add additional messages when reading configuration ||
 +
|}
 +
 +
===Reference/Resource Files===
 +
* See [[GotCloud: Genetic Reference and Resource Files]] for reference/resource file configuration settings
 +
 +
=== Analysis Region Options ===
 +
See [[#Targeted/Exome Sequencing Settings|Targeted/Exome Sequencing Settings]] for more information on specifying exome/targetted regions and other settings.
 +
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --chrs ''# #'' || CHRS || space separated list of chromosomes to process || 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 X
 +
|-
 +
| --region ''#:#-#'' || GENOMESTRIP_REGION || call region - skip regions of chromosome outside of specified region
 +
format (-end is optional): chr:start-end
 +
|
 +
|-
 +
|}
 +
 +
=== Path Options ===
 +
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
 +
|-
 +
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
 +
|-
 +
| --bam-prefix ''prefix'' || BAM_PREFIX || path to prepend to relative BAM file paths in the BAM list ||
 +
|-
 +
| --ref-prefix ''prefix'' || REF_PREFIX || path to prepend to relative reference/resource file paths ||
 +
|-
 +
| --base-prefix ''prefix'' || BASE_PREFIX || path to prepend to relative paths for the BAM list file, BAM (if BAM_PREFIX isn't specified), reference/resource files (if REF_PREFIX isn't specified) ||
 +
|-
 +
| --gotcloudroot ''path'' || GOTCLOUD_ROOT || specify to use a different directory for finding GotCloud bins/scripts || based on the location of the gotcloud/umake.pl script
 +
|-
 +
|}
 +
  
 
== Running GotCloud/GenomeSTRiP ==
 
== Running GotCloud/GenomeSTRiP ==

Latest revision as of 18:47, 10 February 2015

Back to parent: GotCloud

GenomeSTRiP

GenomeSTRiP was developed at the Broad Institute and at the McCarroll Lab at the Harvard Medical School Department of Genetics: http://www.broadinstitute.org/software/genomestrip/

If you use GenomeSTRiP for your research, please cite it: 

Handsaker RE, Korn JM, Nemesh J, McCarroll SA
Discovery and genotyping of genome structural polymorphism by sequencing on a population scale.
Nature genetics 43, 269-276 (2011)
PMID: 21317889

GenomeStrip is currently included within GotCloud and can be run through GotCloud.

Why use GenomeSTRiP?

  1. GenomeSTRiP is a mature software for detecting and genotyping large deletions (and duplications soon to be implemented). In 1000 Genomes, GenomeSTRiP was demonstrated as one of the top-performing SV caller in most evaluation metrics.
  2. GenomeSTRiP is a great tool to integrate across multiple structural variant calls. When multiple structural variant calls exists, all the other variants can be genotyped and filtered with GenomeSTRiP, and that is how 1000 Genomes structural variant call sets were made.
  3. Currently, GenomeSTRiP only allows calling large deletions, but duplicate calling pipeline is under way.

Why do we use GotCloud/GenomeSTRiP pipeline?

  1. The main purpose of GotCloud pipelines is to provide a pipeline for users with limited knowledge and experience with high performance computing environment.
    • GotCloud/GenomeSTRiP provide a simple interface consistent to alignment, SNP, and indel calling.
    • GenomeSTRiP itself also provides a straightforward pipeline to use as standalone software
  2. GotCloud supports a variety of cluster environment that is not currently supported by GenomeSTRiP
    • GenomeSTRiP is designed based on a framework called Qscript, which provide a nice support for LSF cluster system
    • GotCloud support many additional cluster environments such as MOSIX or SLURM we use locally at Michigan.
  3. GotCloud also provide a fault-tolerant solution for large-scale jobs.
    • GotCloud automatically picks up jobs from the point where it failed. This allows easier and simpler run against potential technical glitches in the system.


Overview of GotCloud/GenomeSTRiP pipeline

GotCloud/GenomeSTRiP pipeline consists of three separate steps.

  • Preprocess step : Create metadata summarizing the GC profiles, depth distribution, insert size distribution for accurate discovery and genotyping of structural variants.
  • Discovery step : Perform variant discovery split by region, across all samples. Also, perform variant filtering based on expert knowledge.
  • Genotyping step : Iterate discovered variants across the samples and calculate the genotype likelihood of for each possible genotype.

In addition, if one wants to genotype structural variants from other structural variant caller, there is a step available.

  • Third-party Genotyping and Filtering step : Perform genotyping on the variant sites specified by an input VCF, and also perform variant filtering.


Input Data

Input Data

BAM Files

The BAM files need to be duplicate-marked and base-quality recalibrated in order to obtain high quality SNP calls. Generating these BAM files from original FASTQs is automatically done as part of the Alignment Pipeline of GotCloud.

BAM List File

  • Automatically created when running the GotCloud Alignment Pipeline
  • Each line of the BAM list file represents a single individual

Columns:

  1. sample id
  2. comma separated population labels (optional column)
  3. BAM File 1 (preferable to have full paths to BAM files)
  4. BAM File 2 (if more than 1 BAM per sample)
...
# BAM File N (if more than 1 BAM per sample)
[SAMPLE_ID]    [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...

or 

[SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ...
  • Notes:
    • tab delimited
    • multiple BAMs per individual may be provided, but should all be on the same line of the list file
    • population label is optional - it will default to ALL
      • only used by Thunder (part of ldrefine pipeline)
      • if all samples are from the same population, population label can be skipped or you can just specify ALL for the population label for each sample.

The path to the BAM List file is defaulted to the outputDirectory/bam.list. It can be overridden by setting --bamlist, --bam_list, or --list on the command-line or by setting BAM_LIST in your configuration file to the path to the BAM List File. See Required Options for more information.

Reference Files

See GotCloud: Genetic Reference and Resource Files for detailed information about the multiple required reference files for the variant calling pipeline, including:

  • How to obtain default references
  • Configuration keys & default values
  • How to generate your own references
  • How to point GotCloud to your reference files

Required Reference File Types:

Configuration File

The GotCloud configuration file contains the run-time options, including software binaries and command line arguments. A default configuration file is automatically loaded. Users may specify their own configuration file specifying just the values different than the defaults. The configuration file is not required if there are no values to override.

  • Default GotCloud configuration file is gotcloud/bin/gotcloudDefaults.conf
  • Comments begin with a #
  • Format: KEY = value
    • where KEY is the item being set and value is its new value
  • Some settings can be defined both in the configuration file and on the GotCloud command-line
    • command-line options take priority over configuration file settings
  • A KEY can be used in another KEY's value by specifying $(KEY)
    • Example:
      KEY1 = value1
      KEY2 = $(KEY1)/value2
      • When KEY2 is used, it will be equal to: value1/value2

Output Directory

  • The output directory is required for running GotCloud, so GotCloud knows where to write its output
Configuration Key Command-line Flag Value Description
OUT_DIR --outdir output directory

Reference/Resource Files

Cluster Configuration

Currently GotCloud/GenomeSTRiP only supports mosix and will wait for each job to complete after launching it.

Configuration Key Command-line Flag Value Description
GENOMESTRIP_MOSIX_OPT --mosix-opt options to pass to mosix, example: 
-j36,37,38,39,40,41,45,46,47,48,49

GenomeSTRiP specific configuration settings

When using GenomeSTRiP, you need to specify the following configuration settings: 

GENOMESTRIP_MASK_FASTA = /net/bipolar/hmkang/ref/hs37d5/genomeSTRiP/human_g1k_v37.mask.100.fasta
GENOMESTRIP_PLOIDY_MAP = /net/bipolar/hmkang/2013_09/seqshop/reference/svtoolkit/conf/humgen_g1k_v37_ploidy.map

Replace the specified paths to the path to these files.

GotCloud/GenomeSTRiP Command-line Options/Configuration Settings

Required Options

Command-line Flag Configuration Key Value Description Default Value
--outdir path OUT_DIR output directory
--list file BAM_LIST path to the BAM List File $(OUT_DIR)/bam.list
--numjobs # number of jobs to run in parallel 0 (generate Makefile of steps, but do not run)

Common Options

Common Options
Command-line Flag Configuration Key Value Description Default Value
--conf file configuration file to use

Cluster Options

Command-line Flag Configuration Key Value Description Default Value
--mosix-opt opts GENOMESTRIP_MOSIX_OPT options to pass to the mosix command

Test/Debug Options

Command-line Flag Configuration Key Value Description Default Value
--verbose Add additional messages when reading configuration

Reference/Resource Files

Analysis Region Options

See Targeted/Exome Sequencing Settings for more information on specifying exome/targetted regions and other settings.

Command-line Flag Configuration Key Value Description Default Value
--chrs # # CHRS space separated list of chromosomes to process 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 X
--region #:#-# GENOMESTRIP_REGION call region - skip regions of chromosome outside of specified region

format (-end is optional): chr:start-end

Path Options

Command-line Flag Configuration Key Value Description Default Value
--bam-prefix prefix BAM_PREFIX path to prepend to relative BAM file paths in the BAM list
--ref-prefix prefix REF_PREFIX path to prepend to relative reference/resource file paths
--base-prefix prefix BASE_PREFIX path to prepend to relative paths for the BAM list file, BAM (if BAM_PREFIX isn't specified), reference/resource files (if REF_PREFIX isn't specified)
--gotcloudroot path GOTCLOUD_ROOT specify to use a different directory for finding GotCloud bins/scripts based on the location of the gotcloud/umake.pl script


Running GotCloud/GenomeSTRiP

The general command-line for running GenomeSTRiP via GotCloud is: 

gotcloud genomestrip --run-<step> --conf <gotcloud.conf> --outdir <outputDirectory> --numjobs <#>

Where:

Optional Parameters:

  • --metadata <metadataDirectory> - points to a directory containing pre-made metadata files
    • Only required if skipping the --run-metadata step.

Metadata Pipeline

The metadata pipeline creates metadata summarizing genome-wide statistics such as GC profiles, depth distribution, insert size distributions.

This metadata pipeline runs the "GenomeSTRiP SVProcess" step, generating metadata output and other intermediate files. See http://gatkforums.broadinstitute.org/discussion/1514/svpreprocess-queue-script for the details of the Preprocess step.

NOTE: You don't always have to create the metadata on your own. You can in principle use the public metadata generated for 1000G samples, under the assumption that the metadata share similar characteristics to your samples. But if you have enough computing resources, the best practice is to create metadata specifically for your sequence data.

Timing:

  • 10 BAMs, chr 21 and 22: 1 hour, 20 mins with 10 jobs
  • May take a few weeks to run whole genome on 2,000 BAMs


Discovery Pipeline

The discovery pipeline performs variant discovery across all samples as well as variant filtering based on expert knowledge.

Timing:

  • 10 BAMs, chr 21 and 22: 1 hour, 20 mins with 10 jobs

Genotyping Pipeline

The genotyping pipeline iterates the discovered variants across the samples, calculating the genotype likelihood for each possible genotype.

Timing:

  • 10 BAMs, chr 21 and 22: 4 mins with 10 jobs

3rd-party Site Genotyping/Filtering Pipeline =

The third-party genotyping and filtering pipeline performs genotyping on the variant sites specified by an input (3rd-party site) VCF and also performs variant filtering. 

gotcloud genomestrip --run-thirdparty --in-vcf 1kg.phase1.chr22.36Mb.sites.vcf --conf gotcloud.conf --outdir outputDirectory --numjobs 10
Retrieved from "http://genome.sph.umich.edu/w/index.php?title=GotCloud:_GenomeSTRiP_Pipeline&oldid=12814"