Line 1: |
Line 1: |
− | Back to the beginning [http://genome.sph.umich.edu/wiki/Pipelines]
| |
| | | |
− | The Variant Calling Pipeline (UMAKE) takes recalibrated BAM files and detects SNPs and calls their genotypes, producing VCF files.
| + | Back to parent: [[GotCloud]] |
| | | |
− | == Input Data:==
| + | The Variant Calling Pipeline (previously called 'UMAKE') makes genotype calls from recalibrated BAM files. These genotype calls are output into [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) files]. |
− | *Aligned/Processed/Recalibrated BAM files
| |
− | *Index file containing Sample IDs & BAM file names
| |
− | *Reference files
| |
− | *(Optional) Configuration file to override default options
| |
| | | |
− | === BAM files ===
| |
| | | |
− | === Index file === | + | == Running the GotCloud Variant Calling Pipeline == |
| + | |
| + | The variant calling pipeline (umake) is run using <code>gotcloud snpcall</code> and <code>gotcloud ldrefine</code>. |
| + | |
| + | ===Running the Automatic Test=== |
| + | |
| + | The automatic test runs the variant calling pipeline on a small test set and checks the results against expected results validating that GotCloud is installed correctly. |
| + | |
| + | *Run <code>snpcall</code> pipeline test: |
| + | gotcloud snpcall --test OUTPUT_DIR |
| + | ** Where OUTPUT_DIR is the directory where you want to store the test results |
| + | ** If you see <code>Successfully ran the test case, congratulations!</code>, then you are ready to run snpcall on your own samples. |
| + | *Run <code>ldrefine</code> pipeline test: |
| + | gotcloud ldrefine --test OUTPUT_DIR |
| + | ** Where <code>OUTPUT_DIR</code> is the directory where you want to store the test results |
| + | ** If you see <code>Successfully ran the test case, congratulations!</code>, then you are ready to run ldrefine on your own samples. |
| + | |
| + | == Overview of Variant Calling Pipeline Steps == |
| + | Here is an overview of the Variant Calling Pipeline: |
| + | |
| + | [[File: umakeSteps.png]] |
| + | |
| + | |
| + | For more information on the filters applied during the Variant Calling Pipeline, see, [[GotCloud: Filters]]. |
| + | |
| + | == Input Data== |
| + | * [[#BAM Files|Aligned/Processed/Recalibrated BAM files]] |
| + | * [[#BAM List File|BAM list file containing Sample IDs & BAM file names]] |
| + | * [[#Reference Files|Reference files]] |
| + | * (Optional) [[#Configuration File|Configuration file to override default options]] |
| + | |
| + | === BAM Files === |
| + | The BAM files need to be duplicate-marked and base-quality recalibrated in order to obtain high quality SNP calls. Generating these BAM files from original FASTQs is automatically done as part of the [[Alignment Pipeline]] of GotCloud. |
| + | |
| + | === BAM List File === |
| + | * Automatically created when running the GotCloud [[Alignment Pipeline]] |
| + | * Each line of the BAM list file represents a single individual |
| + | |
| + | Columns: |
| + | # sample id |
| + | # comma separated population labels (optional column) |
| + | # BAM File 1 (preferable to have full paths to BAM files) |
| + | # BAM File 2 (if more than 1 BAM per sample) |
| + | :... |
| + | |
| + | : # BAM File N (if more than 1 BAM per sample) |
| + | [SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ... |
| + | or |
| + | [SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ... |
| + | |
| + | * Notes: |
| + | ** tab delimited |
| + | ** multiple BAMs per individual may be provided, but should all be on the same line of the list file |
| + | ** population label is optional - it will default to <code>ALL</code> |
| + | *** only used by Thunder (part of ldrefine pipeline) |
| + | *** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample. |
| + | |
| + | The path to the BAM List file is defaulted to the <code>outputDirectory/bam.list</code>. It can be overridden by setting <code>--bamlist</code>, <code>--bam_list</code>, or <code>--list</code> on the command-line or by setting BAM_LIST in your configuration file to the path to the BAM List File. See [[#Required_Options|Required Options]] for more information. |
| | | |
| === Reference Files === | | === Reference Files === |
| + | See [[GotCloud: Genetic Reference and Resource Files]] for detailed information about the multiple required reference files for the variant calling pipeline, including: |
| + | * How to obtain default references |
| + | * Configuration keys & default values |
| + | * How to generate your own references |
| + | * How to point GotCloud to your reference files |
| + | |
| + | Required Reference File Types: |
| + | * [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] |
| + | * [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF Files|DBSNP VCF Files]] |
| + | * [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF Files|HapMap3 VCF Files]] |
| + | * [[GotCloud: Genetic Reference and Resource Files#OMNI VCF Files|OMNI VCF Files]] |
| + | * [[GotCloud: Genetic Reference and Resource Files#INDEL VCF File(s)|INDEL VCF File(s)]] |
| | | |
| === Configuration File === | | === Configuration File === |
| + | {{:GotCloud: Configuration}} |
| + | |
| + | See [[#Variant Calling Command-line Options/Configuration Settings|Variant Calling Command-line Options/Configuration Settings]] for more information on Configuration options. |
| + | |
| + | ==== Example Configuration File ==== |
| + | Example configuration file where reference files happen to be stored in /path/reference, and bam index file in path/freeze5 |
| + | CHRS = 20 22 |
| + | BAM_LIST = /path/freeze5.bam.list |
| + | OUT_DIR = /path/freeze5/output |
| + | REF_DIR = /path/reference/ |
| + | REF = $(REF_DIR)/hs37d5.fa |
| + | INDEL_PREFIX = $(REF_DIR)/1kg.pilot_release.merged.indels.sites.hg19 |
| + | HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz |
| + | DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz |
| + | |
| + | |
| + | == Variant Calling Command-line Options/Configuration Settings == |
| + | {{:GotCloud: Variant Calling Options}} |
| + | |
| + | |
| + | == Use Cases & Recommended Settings == |
| + | === Single Sample Processing === |
| + | To run single sample processing we recommend adding the following settings to your configuration file: |
| + | UNIT_CHUNK = 20000000 |
| + | MODEL_GLFSINGLE = TRUE |
| + | MODEL_SKIP_DISCOVER = FALSE |
| + | MODEL_AF_PRIOR = TRUE |
| + | VCF_EXTRACT = $(REF_DIR)/snpOnly.vcf.gz |
| + | EXT = $(REF_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(REF_DIR)/chrCHR.filtered.sites.vcf.gz |
| + | |
| + | Explanation of these settings: |
| + | * <code>UNIT_CHUNK</code> - since this is only 1 sample, process larger regions at a time than default |
| + | * <code>MODEL_GLFSINGLE</code> - single sample, so model glfsingle |
| + | * <code>MODEL_SKIP_DISCOVER</code> - do not skip the variant discovery step |
| + | * <code>MODEL_AF_PRIOR</code> - use AF prior for genotyping |
| + | * <code>VCF_EXTRACT</code> - VCF file to use for extracting the site information to genotype |
| + | ** This file is included in the latest reference release: [[GotCloud:_Genetic_Reference_and_Resource_Files#hs37d5-db142|hs37d5-db142]] |
| + | * <code>EXT</code> - VCF reference files to use for the external filtering |
| + | ** These files are included in the latest reference release: [[GotCloud:_Genetic_Reference_and_Resource_Files#hs37d5-db142|hs37d5-db142]] |
| + | |
| + | |
| | | |
| == Running == | | == Running == |
| | | |
− | Running umake is straightforward: | + | Running variant calling is straightforward: |
| | | |
| <code> | | <code> |
− | '''cd ~/myseq''' | + | '''gotcloud snpcall --conf vc.conf --numjobs 2 |
− | '''/usr/local/biopipe/bin/umake --conf myconf ???'''
| + | '''gotcloud ldrefine --conf vc.conf --numjobs 2 |
− | '''make -f [out-prefix].Makefile -j [# parallel jobs]''' | |
| </code> | | </code> |
| + | |
| + | * Replace <code>vc.conf</code> with the path/name of the user's configuration file |
| + | ** If you are not overriding any defaults, you can alternatively specify <code>--list path/bam.list</code> replacing <code>path/bam.list</code> with the path/name of your BAM list file. |
| + | * Replace <code>2</code> following <code>--numjobs</code> with the number of jobs to be run in parallel |
| + | * If <code>OUT_DIR</code> is not defined in the configuration file, add <code>--outdir</code> followed by the path to the user's desired output directory. |
| + | |
| + | === Running on a Cluster === |
| + | See [[#Cluster Configuration|Cluster Configuration]] for information on how to configure GotCloud to run on a cluster. |
| + | |
| + | == Results == |
| + | |
| + | If there is a failure, you should see a message like: |
| + | make: *** [...] Error 1 |
| + | Where ... is filled in with other text indicating what step failed. |
| + | |
| + | On SNP Call success, you should see the following output sub-directories under your output directory: |
| + | * glfs with a bams & samples subdirectory |
| + | * pvcfs with a subdirectory per chromosome and then per region |
| + | * '''split''' with a subdirectory per chromosome |
| + | * '''vcfs''' with a subdirectory per chromosome |
| + | * (optionally your target directory) |
| + | |
| + | Under the '''vcf/chrXX''' directory, there should be: |
| + | * chrXX.filtered.sites.vcf |
| + | * chrXX.filtered.sites.vcf.norm.log |
| + | * chrXX.filtered.sites.vcf.summary |
| + | * '''chrXX.filtered.vcf.gz''' - final filtered variant call file |
| + | * chrXX.filtered.vcf.gz.OK |
| + | * chrXX.filtered.vcf.gz.tbi |
| + | * chrXX.hardfiltered.sites.vcf |
| + | * chrXX.hardfiltered.sites.vcf.log |
| + | * chrXX.hardfiltered.sites.vcf.summary |
| + | * chrXX.hardfiltered.vcf.gz |
| + | * chrXX.hardfiltered.vcf.gz.OK |
| + | * chrXX.hardfiltered.vcf.gz.tbi |
| + | * chrXX.merged.sites.vcf |
| + | * chrXX.merged.stats.vcf |
| + | * chrXX.merged.vcf |
| + | * chrXX.merged.vcf.OK |
| + | |
| + | The .merged.vcf is the merged together versions of the separate regions in the same chromosome. |
| + | |
| + | The filtered is the merged.vcf after it has been run through filters and is marked with PASS/FAIL. |
| + | |
| + | Under the '''split/chrXX''' directory, there should be: |
| + | * chrXX.filtered.PASS.split.[N].vcf.gz |
| + | * chrXX.filtered.PASS.split.err |
| + | * chrXX.filtered.PASS.split.vcflist |
| + | * '''chrXX.filtered.PASS.gz''' - final variant call file with only PASS variants |
| + | * subset.OK |