Changes

From Genome Analysis Wiki
Jump to navigationJump to search

GotCloud: Variant Calling Pipeline (view source)

Revision as of 18:54, 30 March 2015

4,130 bytes added ,  18:54, 30 March 2015
→‎Running the Automatic Test
Line 1: Line 1: −
Back to the beginning [http://genome.sph.umich.edu/wiki/Pipelines]
     −
The Variant Calling Pipeline (UMAKE) takes recalibrated BAM files and detects SNPs and calls their genotypes, producing VCF files.
+
Back to parent: [[GotCloud]]
   −
== Input Data:==
+
The Variant Calling Pipeline (previously called 'UMAKE') makes genotype calls from recalibrated BAM files. These genotype calls are output into [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) files].
−
*Aligned/Processed/Recalibrated BAM files
  −
*Index file containing Sample IDs & BAM file names
  −
*Reference files
  −
*(Optional) Configuration file to override default options
     −
=== BAM files ===
  −
The BAM files need to be duplicate-marked and base-quality recalibrated in order to obtain high quality SNP calls.
     −
FASTQs can be converted to this type of BAM using the [[Mapping Pipeline]].
+
== Running the GotCloud Variant Calling Pipeline ==
    +
The variant calling pipeline (umake) is run using <code>gotcloud snpcall</code> and <code>gotcloud ldrefine</code>.
   −
Additional input Files including Pedigree files (PED format) (to specify gender information in chrX calling), Target information (UCSC's BED format) in targeted or whole exome capture sequencing may be provided.
+
===Running the Automatic Test===
−
Configuration file contains core information of run-time options including the software binaries and command line arguments. Refer to the example configuration file for further information
+
 
−
[edit]
+
The automatic test runs the variant calling pipeline on a small test set and checks the results against expected results validating that GotCloud is installed correctly.
−
=== Index File ===
+
 
−
Each line of the index file represents each individual under the following format. Note that multiple BAMs per individual may be provided.
+
*Run <code>snpcall</code> pipeline test:
−
[SAMPLE_ID]   [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...
+
gotcloud snpcall --test OUTPUT_DIR
 +
** Where OUTPUT_DIR is the directory where you want to store the test results
 +
** If you see <code>Successfully ran the test case, congratulations!</code>, then you are ready to run snpcall on your own samples.
 +
*Run <code>ldrefine</code> pipeline test:
 +
gotcloud ldrefine --test OUTPUT_DIR
 +
** Where <code>OUTPUT_DIR</code> is the directory where you want to store the test results
 +
** If you see <code>Successfully ran the test case, congratulations!</code>, then you are ready to run ldrefine on your own samples.
 +
 
 +
== Overview of Variant Calling Pipeline Steps ==
 +
Here is an overview of the Variant Calling Pipeline:
 +
 
 +
[[File: umakeSteps.png]]
 +
 
 +
 
 +
For more information on the filters applied during the Variant Calling Pipeline, see, [[GotCloud: Filters]].
 +
 
 +
== Input Data==
 +
* [[#BAM Files|Aligned/Processed/Recalibrated BAM files]]
 +
* [[#BAM List File|BAM list file containing Sample IDs & BAM file names]]
 +
* [[#Reference Files|Reference files]]
 +
* (Optional) [[#Configuration File|Configuration file to override default options]]
 +
 
 +
=== BAM Files ===
 +
The BAM files need to be duplicate-marked and base-quality recalibrated in order to obtain high quality SNP calls. Generating these BAM files from original FASTQs is automatically done as part of the [[Alignment Pipeline]] of GotCloud.
 +
 
 +
=== BAM List File ===
 +
* Automatically created when running the GotCloud [[Alignment Pipeline]]
 +
* Each line of the BAM list file represents a single individual
    
Columns:
 
Columns:
 
# sample id
 
# sample id
−
# comma separated population labels
+
# comma separated population labels (optional column)
−
# BAM File 1
+
# BAM File 1 (preferable to have full paths to BAM files)
−
# BAM File 2 (if applicable)
+
# BAM File 2 (if more than 1 BAM per sample)
 
:...
 
:...
   −
: # BAM File N
+
: # BAM File N (if more than 1 BAM per sample)
 +
[SAMPLE_ID]    [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...
 +
or
 +
[SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ...
 +
 
 +
* Notes:
 +
** tab delimited
 +
** multiple BAMs per individual may be provided, but should all be on the same line of the list file
 +
** population label is optional - it will default to <code>ALL</code>
 +
*** only used by Thunder (part of ldrefine pipeline)
 +
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.
 +
 
 +
The path to the BAM List file is defaulted to the <code>outputDirectory/bam.list</code>.  It can be overridden by setting <code>--bamlist</code>, <code>--bam_list</code>, or <code>--list</code> on the command-line or by setting BAM_LIST in your configuration file to the path to the BAM List File.  See [[#Required_Options|Required Options]] for more information.
    
=== Reference Files ===
 
=== Reference Files ===
−
Reference files are required for doing Variant Calling. 
+
See [[GotCloud: Genetic Reference and Resource Files]] for detailed information about the multiple required reference files for the variant calling pipeline, including:
−
 
+
* How to obtain default references
−
See [[#Reference Files| Configuration Files: Reference Files]] for information on how to specify the reference files in the configuration.
+
* Configuration keys & default values
 +
* How to generate your own references
 +
* How to point GotCloud to your reference files
    +
Required Reference File Types:
 +
* [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]]
 +
* [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF Files|DBSNP VCF Files]]
 +
* [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF Files|HapMap3 VCF Files]]
 +
* [[GotCloud: Genetic Reference and Resource Files#OMNI VCF Files|OMNI VCF Files]]
 +
* [[GotCloud: Genetic Reference and Resource Files#INDEL VCF File(s)|INDEL VCF File(s)]]
    
=== Configuration File ===
 
=== Configuration File ===
−
A default configuration file is automatically loaded.  Users must specify their own configuration file specifying just the values different than the defaults.
+
{{:GotCloud: Configuration}}
   −
Comments begin with a <code>#</code>
+
See [[#Variant Calling Command-line Options/Configuration Settings|Variant Calling Command-line Options/Configuration Settings]] for more information on Configuration options.
   −
Format: KEY = value
+
==== Example Configuration File ====
 +
Example configuration file where reference files happen to be stored in /path/reference, and bam index file in path/freeze5
 +
CHRS = 20 22
 +
BAM_LIST = /path/freeze5.bam.list
 +
OUT_DIR = /path/freeze5/output
 +
REF_DIR = /path/reference/
 +
REF = $(REF_DIR)/hs37d5.fa
 +
INDEL_PREFIX = $(REF_DIR)/1kg.pilot_release.merged.indels.sites.hg19
 +
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz
 +
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz
   −
Where KEY is the item being set and value is its new value
      +
== Variant Calling Command-line Options/Configuration Settings ==
 +
{{:GotCloud: Variant Calling Options}}
   −
====Required User Config Files Settings====
  −
The following Config File Settings must be specified by the user:
  −
* CHRS = # space separated list of chromosomes you want
  −
* BAM_INDEX = # path to the Index File of BAMs
     −
====Required on Command-Line or in Config File====
+
== Use Cases & Recommended Settings ==
−
The following Command-Line or Config File Settings must be specified by the user:
+
=== Single Sample Processing ===
−
* --outdir/OUTDIR= # path to desired output directory
+
To run single sample processing we recommend adding the following settings to your configuration file:
 +
UNIT_CHUNK = 20000000
 +
MODEL_GLFSINGLE = TRUE
 +
MODEL_SKIP_DISCOVER = FALSE
 +
MODEL_AF_PRIOR = TRUE
 +
VCF_EXTRACT = $(REF_DIR)/snpOnly.vcf.gz
 +
EXT = $(REF_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(REF_DIR)/chrCHR.filtered.sites.vcf.gz
   −
====Targeted/Exome Sequencing Settings====
+
Explanation of these settings:
−
If you are running Targeted/Exome Sequencing, the user should specify:
+
* <code>UNIT_CHUNK</code> - since this is only 1 sample, process larger regions at a time than default
−
* Write loci file when performing pileup
+
* <code>MODEL_GLFSINGLE</code> - single sample, so model glfsingle
−
** WRITE_TARGET_LOCI = TRUE
+
* <code>MODEL_SKIP_DISCOVER</code> - do not skip the variant discovery step
−
* Specify the directory to store target information, for example: targetDir
+
* <code>MODEL_AF_PRIOR</code> - use AF prior for genotyping
−
** TARGET_DIR = targetDir
+
* <code>VCF_EXTRACT</code> - VCF file to use for extracting the site information to genotype
 +
** This file is included in the latest reference release: [[GotCloud:_Genetic_Reference_and_Resource_Files#hs37d5-db142|hs37d5-db142]]
 +
* <code>EXT</code> - VCF reference files to use for the external filtering
 +
** These files are included in the latest reference release: [[GotCloud:_Genetic_Reference_and_Resource_Files#hs37d5-db142|hs37d5-db142]]
   −
If all individuals have the same target:
  −
* Specify the single bed file, for example: target.bed
  −
** UNIFORM_TARGET_BED = target.bed
     −
If not all individuals have the same target:
  −
* Specify the file containing the sample id -> bed map, for example: targetMap.txt
  −
** MULTIPLE_TARGET_MAP = targetMap.txt
  −
*** Each line of the file contains [SM_ID] [TARGET_BED]
     −
Optional Settings:
+
== Running ==
−
* Extend the target region by a given number of bases, for example: 50
  −
** OFFSET_OFF_TARGET = 50
  −
*  Exclude off-target regions when using samtools view (may make command line too long)
  −
** SAMTOOLS_VIEW_TARGET_ONLY = TRUE
      +
Running variant calling is straightforward:
   −
==== Reference Files ====
+
<code>
−
* Reference Sequence in fasta format.
+
'''gotcloud snpcall --conf vc.conf --numjobs 2
−
** REF = path/file.fa
+
'''gotcloud ldrefine --conf vc.conf --numjobs 2
−
* Indel VCF File Prefix
+
</code>
−
** INDEL_PREFIX = path/indels.sites.hg19
  −
** path/ contains indels.sites.hg19.chr20.vcf for each chromosome being processed
  −
* DBSNP File Prefix
  −
** DBSNP_PREFIX = path/dbsnp_135_b37.rod
  −
** path/ contains dbsnp_135_b37.rod.chr20.map for each chromosome being processed
  −
* HapMap3 polymorphic site prefix
  −
** HM3_PREFIX = path/hapmap3.qc.poly
  −
** path/ contains hapmap3.qc.poly.chr20.bim & hapmap3.qc.poly.chr20.frq for each chromosome being processed
     −
Can be downloaded from: [[ftp://share.sph.umich.edu/1000genomes/umake-resources/ | FTP Download of Full Resource Files]]
+
* Replace <code>vc.conf</code> with the path/name of the user's configuration file
 +
** If you are not overriding any defaults, you can alternatively specify <code>--list path/bam.list</code> replacing <code>path/bam.list</code> with the path/name of your BAM list file.
 +
* Replace <code>2</code> following <code>--numjobs</code> with the number of jobs to be run in parallel
 +
* If <code>OUT_DIR</code> is not defined in the configuration file, add <code>--outdir</code> followed by the path to the user's desired output directory.
   −
INDEL_PREFIX = $(UMAKE_ROOT)/ref/indels/1kg.pilot_release.merged.indels.sites.hg19 # 1000 Genomes Pilot 1 indel VCF prefix
+
=== Running on a Cluster ===
−
DBSNP_PREFIX =  $(UMAKE_ROOT)/ref/dbSNP/dbsnp_135_b37.rod # dbSNP file prefix
+
See [[#Cluster Configuration|Cluster Configuration]] for information on how to configure GotCloud to run on a cluster.
−
HM3_PREFIX =  $(UMAKE_ROOT)/ref/HapMap3/hapmap3_r3_b37_fwd.consensus.qc.poly # HapMap3 polymorphic site prefix
     −
==== Chromosome X Calling ====
+
== Results ==
−
* PED_INDEX = pedfile.ped
      +
If there is a failure, you should see a message like:
 +
make: *** [...] Error 1
 +
Where ... is filled in with other text indicating what step failed.
   −
== Running ==
+
On SNP Call success, you should see the following output sub-directories under your output directory:
 +
* glfs with a bams & samples subdirectory
 +
* pvcfs with a subdirectory per chromosome and then per region
 +
* '''split''' with a subdirectory per chromosome
 +
* '''vcfs''' with a subdirectory per chromosome
 +
* (optionally your target directory)
 +
 
 +
Under the '''vcf/chrXX''' directory, there should be:
 +
* chrXX.filtered.sites.vcf
 +
* chrXX.filtered.sites.vcf.norm.log
 +
* chrXX.filtered.sites.vcf.summary
 +
* '''chrXX.filtered.vcf.gz''' - final filtered variant call file
 +
* chrXX.filtered.vcf.gz.OK
 +
* chrXX.filtered.vcf.gz.tbi
 +
* chrXX.hardfiltered.sites.vcf
 +
* chrXX.hardfiltered.sites.vcf.log
 +
* chrXX.hardfiltered.sites.vcf.summary
 +
* chrXX.hardfiltered.vcf.gz
 +
* chrXX.hardfiltered.vcf.gz.OK
 +
* chrXX.hardfiltered.vcf.gz.tbi
 +
* chrXX.merged.sites.vcf
 +
* chrXX.merged.stats.vcf
 +
* chrXX.merged.vcf
 +
* chrXX.merged.vcf.OK
 +
 
 +
The .merged.vcf is the merged together versions of the separate regions in the same chromosome.
   −
Running umake is straightforward:
+
The filtered is the merged.vcf after it has been run through filters and is marked with PASS/FAIL.
   −
<code>
+
Under the '''split/chrXX''' directory, there should be:
−
'''cd ~/myseq'''
+
* chrXX.filtered.PASS.split.[N].vcf.gz
−
'''/usr/local/biopipe/bin/umake --conf myconf ???'''
+
* chrXX.filtered.PASS.split.err
−
'''make -f [out-prefix].Makefile -j [# parallel jobs]'''
+
* chrXX.filtered.PASS.split.vcflist
−
</code>
+
* '''chrXX.filtered.PASS.gz''' - final variant call file with only PASS variants
 +
* subset.OK
Kleckner
87

edits

Retrieved from "http://genome.sph.umich.edu/wiki/Special:MobileDiff/5740...13079"

Navigation menu