GotCloud: GenomeSTRiP Pipeline
Back to parent: GotCloud
GenomeSTRiP
GenomeSTRiP was developed at the Broad Institute and at the McCarroll Lab at the Harvard Medical School Department of Genetics: http://www.broadinstitute.org/software/genomestrip/
If you use GenomeSTRiP for your research, please cite it:
Handsaker RE, Korn JM, Nemesh J, McCarroll SA Discovery and genotyping of genome structural polymorphism by sequencing on a population scale. Nature genetics 43, 269-276 (2011) PMID: 21317889
GenomeStrip is currently included within GotCloud and can be run through GotCloud.
Why use GenomeSTRiP?
- GenomeSTRiP is a mature software for detecting and genotyping large deletions (and duplications soon to be implemented). In 1000 Genomes, GenomeSTRiP was demonstrated as one of the top-performing SV caller in most evaluation metrics.
- GenomeSTRiP is a great tool to integrate across multiple structural variant calls. When multiple structural variant calls exists, all the other variants can be genotyped and filtered with GenomeSTRiP, and that is how 1000 Genomes structural variant call sets were made.
- Currently, GenomeSTRiP only allows calling large deletions, but duplicate calling pipeline is under way.
Why do we use GotCloud/GenomeSTRiP pipeline?
- The main purpose of GotCloud pipelines is to provide a pipeline for users with limited knowledge and experience with high performance computing environment.
- GotCloud/GenomeSTRiP provide a simple interface consistent to alignment, SNP, and indel calling.
- GenomeSTRiP itself also provides a straightforward pipeline to use as standalone software
- GotCloud supports a variety of cluster environment that is not currently supported by GenomeSTRiP
- GenomeSTRiP is designed based on a framework called Qscript, which provide a nice support for LSF cluster system
- GotCloud support many additional cluster environments such as MOSIX or SLURM we use locally at Michigan.
- GotCloud also provide a fault-tolerant solution for large-scale jobs.
- GotCloud automatically picks up jobs from the point where it failed. This allows easier and simpler run against potential technical glitches in the system.
Overview of GotCloud/GenomeSTRiP pipeline
GotCloud/GenomeSTRiP pipeline consists of three separate steps.
- Preprocess step : Create metadata summarizing the GC profiles, depth distribution, insert size distribution for accurate discovery and genotyping of structural variants.
- Discovery step : Perform variant discovery split by region, across all samples. Also, perform variant filtering based on expert knowledge.
- Genotyping step : Iterate discovered variants across the samples and calculate the genotype likelihood of for each possible genotype.
In addition, if one wants to genotype structural variants from other structural variant caller, there is a step available.
- Third-party Genotyping and Filtering step : Perform genotyping on the variant sites specified by an input VCF, and also perform variant filtering.
Input Data
Configuration File
The GotCloud configuration file contains the run-time options, including software binaries and command line arguments. A default configuration file is automatically loaded. Users may specify their own configuration file specifying just the values different than the defaults. The configuration file is not required if there are no values to override.
- Default GotCloud configuration file is
gotcloud/bin/gotcloudDefaults.conf
- Comments begin with a
#
- Format:
KEY = value
- where
KEY
is the item being set andvalue
is its new value
- where
- Some settings can be defined both in the configuration file and on the GotCloud command-line
- command-line options take priority over configuration file settings
- A KEY can be used in another KEY's value by specifying $(KEY)
- Example:
KEY1 = value1
KEY2 = $(KEY1)/value2
- When
KEY2
is used, it will be equal to:value1/value2
- Example:
Output Directory
- The output directory is required for running GotCloud, so GotCloud knows where to write its output
Configuration Key | Command-line Flag | Value Description | ||
---|---|---|---|---|
OUT_DIR | --outdir | output directory |
Reference/Resource Files
- See GotCloud: Genetic Reference and Resource Files for reference/resource file configuration settings
Cluster Configuration
Regardless of the type of cluster system used, GotCloud will wait for each job to complete after launching it.
- For any BATCH_TYPEs that run in batch mode, GotCloud generates a script that will wait until the step is complete before returning
- In a sense, it "fakes" interactive mode for all batch types since it will not proceed until a command is finished
- If you are at UM and are using flux, you can specify either
flux
orpbs
Configuration Key | Command-line Flag | Value Description | ||
---|---|---|---|---|
BATCH_TYPE | --batchtype | type of cluster system | ||
Valid Values | Command to Launch | Command to Check for Completion | ||
mosix |
mosbatch -E/tmp |
N/A - interactive type | ||
sge |
qsub |
qstat -u $USER
| ||
sgei |
qrsh -now n |
N/A - interactive type | ||
pbs |
qsub |
qstat -u $USER
| ||
slurm |
sbatch |
squeue -u $USER
| ||
slurmi |
|
N/A - interactive type | ||
local |
N/A - local command | N/A - interactive type | ||
BATCH_OPTS | --batchopts | options to pass to your cluster type, example:
-j36,37,38,39,40,41,45,46,47,48,49 |
GenomeSTRiP specific configuration settings
When using GenomeSTRiP, you need to specify the following configuration settings:
GENOMESTRIP_MASK_FASTA = /net/bipolar/hmkang/ref/hs37d5/genomeSTRiP/human_g1k_v37.mask.100.fasta GENOMESTRIP_PLOIDY_MAP = /net/bipolar/hmkang/2013_09/seqshop/reference/svtoolkit/conf/humgen_g1k_v37_ploidy.map
Replace the specified paths to the path to these files.
Running GotCloud/GenomeSTRiP
Metadata Pipeline
The metadata pipeline creates metadata summarizing genome-wide statistics such as GC profiles, depth distribution, insert size distributions.
This metadata pipeline runs the "GenomeSTRiP SVProcess" step, generating metadata output and other intermediate files. See http://gatkforums.broadinstitute.org/discussion/1514/svpreprocess-queue-script for the details of the Preprocess step.
NOTE: You don't always have to create the metadata on your own. You can in principle use the public metadata generated for 1000G samples, under the assumption that the metadata share similar characteristics to your samples. But if you have enough computing resources, the best practice is to create metadata specifically for your sequence data.
Command-line to run the metadata step:
gotcloud genomestrip --run-metadata --conf gotcloud.conf --outdir outputDirectory --numjobs 10
Timing:
- 10 BAMs, chr 21 and 22: 1 hour, 20 mins with 10 jobs
- May take a few weeks to run whole genome on 2,000 BAMs
Discovery Pipeline
The discovery pipeline performs variant discovery across all samples as well as variant filtering based on expert knowledge.
gotcloud genomestrip --run-discovery --conf gotcloud.conf --outdir outputDirectory --numjobs 10
Timing:
- 10 BAMs, chr 21 and 22: 1 hour, 20 mins with 10 jobs
Genotyping Pipeline
The genotyping pipeline iterates the discovered variants across the samples, calculating the genotype likelihood for each possible genotype.
gotcloud genomestrip --run-genotype --conf gotcloud.conf --outdir outputDirectory --numjobs 10
Timing:
- 10 BAMs, chr 21 and 22: 4 mins with 10 jobs
3rd-party Site Genotyping/Filtering Pipeline =
The third-party genotyping and filtering pipeline performs genotyping on the variant sites specified by an input (3rd-party site) VCF and also performs variant filtering.
gotcloud genomestrip --run-thirdparty --in-vcf 1kg.phase1.chr22.36Mb.sites.vcf --conf gotcloud.conf --outdir outputDirectory --numjobs 10