Genome Analysis Wiki - User contributions [en]

GotCloud

2021-09-11T21:23:16Z

Hmkang: /* Getting Help with GotCloud */

__TOC__

== Genomes on the Cloud (GotCloud) Introduction==

To handle the increasing volume of next generation sequencing and genotyping data, we created and developed software pipelines called '''Genomes on the Cloud (GotCloud).'''

GotCloud contains Mapping & Variant Calling Pipelines.

Key Features:
* Connects sequence analysis tools together in automated pipeline
** Alignment, quality control, variant calling
* Robust against unexpected system failure using GNU make
** easy restart after failure
* Massively parallel, can run hundreds of jobs
** Splits large jobs into many pieces
** Simplifies running on clusters
* Scalable to tens of thousands of samples
* Easy to use - Automates series of configurable steps
** user doesn't have to understand/configure/know the many tools required to create high quality results
* Available on Amazon Web Services (AWS) Elastic Compute Cloud (EC2)
* Run on local machines/clusters

GotCloud incorporates the alignment and variant calling pipelines that we have been using at UM into one easy to use, publicly available tool. GotCloud can run on a user's computer, on an instance in a
compute cloud, and/or can split the work up onto a cluster of machines or instances.

[[File:Gotcloud.puzzles.v2.png|500px]]

=== Getting Help with GotCloud ===

Please join in the [http://groups.google.com/group/GotCloud GotCloud Google Group] to ask / discuss / comment about these pipelines.

Currently the "join" button appears to be missing. Click "NEW TOPIC", then select "Join this group". You can then cancel the message post (or post a message).

See [[GotCloud: FAQs]] if you have any questions. If your questions are not answered there, ask questions in the [https://github.com/statgen/gotcloud GotCloud GitHub repository]

=== Sequence Analysis Background Information ===

There are many essential steps in the analysis of next generation sequence data.

Next generation sequence data analysis starts with [http://en.wikipedia.org/wiki/FASTQ_format FASTQ files], the typical format provided from your sequencing center containing the sequence & base quality information for your data.

The fastq files are processed using the [[GotCloud: Alignment Pipeline|alignment pipeline]] which finds the most likely genomic location for each read and stores that information in a [[BAM|BAM (Binary Sequence Alignment/Map format) file]]. In addition to the sequence and base quality information contained in FASTQ files, a BAM file also contains the genomic location and some additional information about the mapping. As part of the [[GotCloud: Alignment Pipeline|alignment pipeline]], the base qualities are adjusted to more accurately reflect the likelihood that the base is correct.

The [[GotCloud: Alignment Pipeline|alignment pipeline]] can be skipped if you already have Deduped and Recalibrated BAM files. If you have BAMs, but they needed to be deduped and recalibrated, you can use our [[GotCloud:_Alignment_Sub-Pipelines#recabQC_2|recabQC pipeline]].

The [[GotCloud: Variant Calling Pipeline|variant calling pipeline]] processes the deduped and recalibrated BAM files produced by the alignment pipeline or that you provide it, generating an initial list of polymorphic sites and genotypes stored in a [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) file]. The [[GotCloud: Variant Calling Pipeline|variant calling pipeline]] then filters the variants using both hard filters and a [[SVM Filtering|Support Vector Machine (SVM)]]. It then uses haplotype information to refine these genotypes in an updated VCF file.

After completing the GotCloud Variant Calling Pipeline, [[EPACTS|EPACTS (Efficient and Parallelizable Association Container Toolbox)]] can be used to perform statistical tests to identify genome-wide association from sequence data.

[[File:GotCloudDiagram.jpg|500px]]

== Publication ==
If you use GotCloud, please cite our publication:
[http://genome.cshlp.org/content/early/2015/04/14/gr.176552.114.abstract Jun, Goo, et al. "An efficient and scalable analysis framework for variant extraction and refinement from population scale DNA sequence data." Genome research (2015): gr-176552.]

== GotCloud Setup ==

You may run the GotCloud software in several modes:
* On your own hardware running Ubuntu or Redhat/CentOS. See the instructions about installing the software below.
* On an Amazon Elastic Compute Cloud using Ubuntu or Redhat/CentOS if you do not have your own set of machines.
** See [[GotCloud: Amazon]] for more information.
** You can run on an EC2 cluster instance created by StarCluster.

GotCloud has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS and Red Hat 6.6. While it should work on other Linux systems, they have not yet been tested.

=== GotCloud on Amazon ===
You can take advantage of GotCloud AMI's when running on Amazon. The GotCloud already includes GotCloud and default reference files.

See [[GotCloud: Amazon]] for instructions on using GotCloud on Amazon.

=== GotCloud Setup on Any Linux Machine ===

==== GotCloud Dependencies ====

GotCloud requires certain things to be installed in order to run:
* perl - gotcloud is a perl script and it calls many other perl scripts
** Zlib.pm - required for perl scripts to read compressed files.
* make - GNU make is used to run the pipelines
* java - required to run the beagle step of the ld-aware genotype refinement
* curses/ncurses (required for samtools)
** On Ubuntu: <code>sudo apt-get install libncurses5 libncurses5-dev</code>
* cmake (required for premo)
** On Ubuntu: <code>sudo apt-get install cmake</code>

You can check if your system has the proper software installed by invoking the command
[gotcloud_path]/scripts/check_requirements.sh

==== Install GotCloud Software ====

You can install gotCloud on your system as (follow the links for the appropriate instructions):
* [[GotCloud: Source Releases|source release]] - contains the scripts and uncompiled source
* [[GotCloud: Binary Releases|binary release]] - contains the scripts and pre-compiled binaries (no source)

[[GotCloud: Versions]] describes the changes added to each version.

Alternatively if you are using Amazon EC2, you can use one of the following sets of instructions:
* Create a machine instance based on the AMI we provide: [[Amazon Single Node]]
* Create an EC2 cluster instance using [[StarCluster|StarCluster]]
For more information on using GotCloud on Amazon, see: [[GotCloud: Amazon]]

For more information on Amazon Web Services, see: https://aws.amazon.com/

==== GotCloud Reference/Resource Files ====
In order to run gotCloud, you need to provide Genetic Reference and Resource Files.

These include information about the reference sequence and dbnsp positions.

See: [[GotCloud: Genetic Reference and Resource Files]] for information about the required files. It contains a description of the required files, information about generating your own versions, as well as a downloadable set of files.
* When running on Amazon, a default set of reference files are included in the GotCloud AMI.

==== Configure GotCloud ====
* [[Configure GotCloud|Configure Gotcloud]] for your installation

== Running GotCloud Software ==

* [[GotCloud: Alignment Pipeline|Alignment Pipeline]]
** [[GotCloud: Alignment Sub-Pipelines|Alignment Sub-Pipelines]] - for if you do not want to run the entire Alignment Pipeline
* [[GotCloud: Variant Calling Pipeline|Variant Calling Pipeline]]
* Indel Calling Pipeline
* [[GotCloud: GenomeSTRiP Pipeline|GenomeSTRiP Pipeline]] (Structural Variation)
* MEI Calling Pipeline - ''Ask if you're interested''

You can also create your own pipelines. Instructions are here:
* [[GotCloud: Creating a New Pipeline]]

=== GotCloud Demos ===
GotCloud Demos (originally from our sequencing workshop):
* [[SeqShop: Sequence Mapping and Assembly Practical]]
* [[SeqShop: Variant Calling and Filtering for SNPs Practical]]
* [[SeqShop: Variant Calling and Filtering for INDELs Practical]]
* [[SeqShop: Analysis of Structural Variation Practical]]

GotCloud on Amazon Demo (snpcall & indel):
* [[GotCloud: Amazon Demo]]

Deprecated: [[Tutorial: GotCloud]]

== UMich Development/Release How-To Notes ==
* [[Releasing GotCloud]]
* Amazon EC2
** [[Creating an AMI on EC2]]
** [[Creating a Snapshot on EC2]] (deprecated)
** [[Mount S3 Volume]]
** Notes on sequence data preparation in [[Amazon Storage|Amazon Storage]].

* [[Git_FAQs#Subtrees|Upgrade Git Subtree]]

Samtools-hybrid

2021-09-11T21:22:19Z

Hmkang:

= What is samtools-hybrid?=
samtools-hybrid is a modified version of samtools.

This hybrid version of samtools is essentially samtools version 0.1.7a (r510), except the 0.1.7a bgzf logic and "calmd" functionality have been replaced with the bgzf and "calmd" from samtools revision r983.

The original samtools-hybrid merged in version 0.1.12a (r862), but it has since been upgraded to r983 to bring in the enhanced BAQ logic.

This hybrid version does not provide the samtools "mpileup" SNP calling capability, nor the "idxstats" and "reheader" capabilities that are present in the full version 0.1.12a and later.

== Why was samtools-hybrid created? ==
The purpose is to provide in one executable Heng Li's BAQ base call quality score adjustment functionality from "calmd", as well as to output valid .glf format when using 'samtools pileup -g'.

Some of our tools require both of these methods and since there is not a version of samtools that supports both pileup generating GLF files and the BAQ logic, we created this hybrid version so we could release single version of samtools for use by our tools.

== Where can I get samtools-hybrid? ==
That code can be downloaded at: https://github.com/statgen/samtools-0.1.7a-hybrid

Alternatively, you can download the file here: [[Media:samtools-0.1.7a-hybrid.v1.1.1.tgz‎|samtools-0.1.7a-hybrid.v1.1.1.tgz]] - Released 7/25/2013

= Troubleshooting =
If you have additional questions or comments, please raise an issue in the [https://github.com/statgen/samtools-0.1.7a-hybrid Github repository].

; I get a 'floating point exception' with a command like <code>samtools-hybrid pileup $bam -g -f $fasta > $bam.glf</code>
: One potential issue is that the fasta index file (.fai file) is either not correct for the fasta file or it is not in the exact format that samtools-hybrid expects. Try removing the .fai file and let samtools-hybrid regenerate the file. That will ensure that samtools-hybrid is reading the file in the format it expects.

C++ Library: libStatGen

2021-09-11T21:21:15Z

Hmkang: /* Copyrights */

[[Category:C++]]
[[Category:libStatGen]]

= Description =
Open source, freely available (GPL license), easy to use C++ APIs
* General Operation Classes including:
** File/Stream I/O – uncompressed, BGZF, GZIP, stdin, stdout
** String processing
** Parameter Parsing
* '''Statistical Genetic Specific Classes''' including:
**Handling Common file formats – SAM/BAM, FASTQ, GLF, VCF (coming soon)
***Accessors to get/set values
***Indexed access to BAM files
**Utility classes, including:
***Cigar – interpretation and mapping between query and reference
***Pileup – structured access to data by individual reference position

Can be used to create your own C++ programs.

Currently the repository is recommended for Unix/Linux users with access to the GNU C++ compiler.

= Copyrights =
Here are links to the copyrights for our code and some of the utilities it uses:
*[https://github.com/statgen/libStatGen/blob/master/general/COPYING GNU GENERAL PUBLIC LICENSE] and [https://github.com/statgen/libStatGen/blob/master/general/LICENSE.txt Our Copyright Note]
*[https://github.com/statgen/libStatGen/blob/master/general/LICENSE.twister Copyright for MERSENNE TWISTER (used in Random.cpp)]
*[https://github.com/statgen/libStatGen/blob/master/samtools/COPYING Samtools Copyright (MIT License)]
Copies of these can be found in our library under libStatGen/copyrights/.

= Join in libStatGen mailing list =

Please join in the [http://groups.google.com/group/libStatGen libStatGen Google Group] to ask / discuss / comment about this library.

= Troubleshooting =
If you are having trouble compiling any of the versions, check [[libStatGen Troubleshooting]] for help. If that does not solve your problem, email me for support.

= Where to Find It =

{{ToolGitRepo|repoName=libStatGen|libStatGen=true|libBaseName=libStatGen}}

== Releases ==
Released Versions are documented at [[libStatGen Download]]

= What has changed =
The <code>pipeline</code> and <code>statgen</code> repositories have been deprecated, so please update to our new framework.

<code>libStatGen</code> is the new git repository for our library code.

There are now separate repositories for specific tools/groups of tools, allowing us to track everything separately so it is easier to follow changes that impact a specific tool or the library in general.

= Library Documentation =
Latest Doxygen documentation:


[http://csg.sph.umich.edu//mktrost/doxygen/current/ Current Library Documentation in Doxygen]

Additional documentation:
* [[libStatGen: general]] - General classes for file processing and performing common tasks (used by most other libraries).
* [[libStatGen: BAM]] - Classes specific for reading/writing/analyzing SAM/BAM files.
* [[libStatGen: GLF]] - Classes specific for reading/writing/analyzing GLF files.
* [[libStatGen: FASTQ]] - Classes specific for reading/writing/analyzing FastQ files.
* [[libStatGen: ASP]] - Classes specific for reading/writing/analyzing ASP files.
* [[libStatGen: VCF]] - Classes specific for reading/writing/analyzing VCF files.

= Using the Library =
== Dependencies ==
* This software requires the following to be installed:
** g++
** development version of zlib (zlib1g-dev on ubuntu)
* Compiles on Linux/Unix

== Building the Library ==

If you type make help, you get the build options.
<pre>
Makefile help
-------------
Type... To...
make Compile opt
make help Display this help screen
make all Compile everything (opt, debug, & profile)
make opt Compile optimized
make debug Compile for debug
make profile Compile for profile
make clean Delete temporary files
make test Execute tests (if there are any)
</pre>

When you just type make, it will by default to make opt (optimized).

Make all indicates opt, debug, and profile.

opt creates <code>libStatGen.a</code>, debug creates <code>libStatGen_debug.a</code>, profile creates <code>libStatGen_profile.a</code>

These libraries are created in the top level libStatGen directory and can then be linked to appropriately for building tools as optimized, debugging, and/or profiling.

== Navigating the Library Subdirectories ==
Under the main libStatGen repository, there are:
*bam - library code for operating on bam files.
*copyrights - copyrights for the library and any code included with it.
*fastq - library code for operating on fastq files.
*general - library code for general operations
*glf - library code for operating on glf files.
*include - after compiling, the library headers are linked here
*Makefiles - directory containing Makefiles that are used in the library and can be used for developing programs using the library
*samtools - library code used from samtools

After Compiling: libStatGen.a, libStatGen_debug.a, libStatGen_profile.a are created at the top level.

=== bam, fastq, general, glf, samtools ===
Object files are placed in an obj directory under each subdirectory with debug & profile objects in obj/debug and obj/profile.

Most also have a test directory. Tests are executed by running <code>make test</code>

=== Makefiles ===
This directory contains base makefiles and makefile settings that are used by the library and by programs being written to use the library.

== Using the Library in Your Own Program ==

=== Starting from a Sample Program (Recommended) ===
[https://github.com/statgen/SampleProgram https://github.com/statgen/SampleProgram] is a simple program demonstrating how to write a tool that uses libStatGen and can be used as a starting point for your tool.

SampleProgram has 4 subdirectories:
* copyrights - contains the copyright information, add your own copyrights as necessary
* obj - this directory is where the object files are placed when the code is compiled (with a subdirectory for debug and profile objects)
* src - this is where your own program code goes
* test - this is where your test code goes. Test code can be setup to run with <code>make test</code> to ensure the program works properly.

'''Using SampleProgram as a starting point for your tool:'''
# Copy SampleProgram into a directory with your program name (it is the starting point for your own program).
# Update ChangeLog, .gitignore, and README.txt as appropriate.
# Add any necessary copyrights to the copyrights directory.
#* No changes to Makefile should be necessary.
# Update Makefile.inc
## Update the VERSION as necessary.
## Replace all occurrences of <code>SAMPLE_PROGRAM</code> with an all caps name for your program.
##* You can then use the <code>LIB_PATH_<your program name></code> environment variable to specify an alternate path to libStatGen specific for your program. In most cases you will not need to do this.
#* No other updates to Makefile.inc should be necessary.
# Add your program (cpp & h files) to the <code>src</code> directory.
# Update src/Makefile
## Set EXE to your program executable (replacing sampleProgram)
## Set TOOLBASE, SRCONLY, and HDRONLY as appropriate for specifying your program file names.
## Set any of the other optional settings as specified in the sample makefile.
#* No other changes should be necessary to src/Makefile.
# Add your tests to the <code>test</code> directory.
# Update test/Makefile as appropriate for specifying how to compile/run your tests.

After compiling a <code>bin</code> directory is created in the top level directory. Your executable goes in there. If you build for <code>debug</code> and/or <code>profile</code>, subdirectories for those are created under <code>bin/</code> and <code>obj</code>.

=== Working from Scratch ===
When compiling your code, be sure to include the library header files found in libStatgen/include/ and link in the appropriate library (opt: libStatGen.a, debug: libStatGen_debug.a, or profile: libStatGen_profile.a).

=== Starting from a Sample Set of Tools ===
[https://github.com/statgen/SampleTools https://github.com/statgen/SampleTools] is a repository containing multiple programs within one directory structure. It demonstrates how to have subdirectories for each tool using libStatGen and can be used as a starting point for your set of tools.

SampleTools has 3 subdirectories:
* copyrights - contains the copyright information, add your own copyrights as necessary
* SampleProgram1 - a dummy demo program to show the structure for having multiple programs
* SampleProgram2 - a second dummy demo program to show the structure for having multiple programs

SampleProgram1 & SampleProgram2 have 2 subdirectories:
* src - this is where your own program code goes
* test - this is where your test code goes. Test code can be setup to run with <code>make test</code> to ensure the program works properly.

Upon compiling, an <code>obj</code> directory is created under <code>SampleProgram1</code> and <code>SampleProgram2</code> and a <code>bin</code> directory is created at the top level. If you build for <code>debug</code> and/or <code>profile</code>, subdirectories for those are created under <code>bin/</code> and <code>SampleProgram1(2)/obj</code>.

'''Using SampleTools as a starting point for your set of tools:'''
# Copy <code>SampleTools</code> into a directory with your toolset name (it is the starting point for your own set of tools).
# Update <code>ChangeLog</code>, <code>.gitignore</code>, and <code>README.txt</code> as appropriate.
# Add any necessary copyrights to the copyrights directory.
# Rename the <code>SampleProgram1</code> and <code>SampleProgram2</code> directories
# Create any additional directories as necessary.
#* Recursively copy the structure/Makefiles from <code>SampleProgram1</code>.
# Update <code>SUBDIRS</code> in <code>Makefile</code> as necessary.
# Update <code>Makefile.inc</code>
## Update the <code>VERSION</code> as necessary.
## Replace all occurrences of <code>SAMPLE_PROGRAM</code> with an all caps name for your toolset.
##* You can then use the <code>LIB_PATH_<your toolset name></code> environment variable to specify an alternate path to libStatGen specific for your program. In most cases you will not need to do this.
#* No other updates to <code>Makefile.inc</code> should be necessary.
# For each Program you want to add:
## Move into the appropriate subdirectory.
##* No change should be made to the program's <code>Makefile</code>
## Add your program (cpp & h files) to the <code>src</code> subdirectory.
## Update src/Makefile
### Set EXE to your program executable (replacing sampleProgram)
### Set TOOLBASE, SRCONLY, and HDRONLY as appropriate for specifying your program file names.
### Set any of the other optional settings as specified in the sample makefile.
##* No other changes should be necessary to src/Makefile.
## Add your tests to the <code>test</code> directory.
## Update test/Makefile as appropriate for specifying how to compile/run your tests.

= How To Use the APIs =
More coming soon, see: http://genome.sph.umich.edu/wiki/Sam_Library_Usage_Examples

[[LibStatGen: ASP#API for Reading ASP Files| ASP APIs]]

[[LibStatGen: VCF#API for Reading VCF Files| VCF APIs]]

Software

2021-09-11T21:20:33Z

Hmkang:

[[Category:Software]]
[[Category:C++]]

=Software=
Due to increasing volume of next generation sequencing and genotyping data, we have created these C++ library and tools that use that library.

This page points to downloads, documentation, and papers for software that is written here at the [http://genome.sph.umich.edu Center for Statistical Genetics]

If you have any questions or comments, please raise issue in [https://github.com/statgen/ our GitHub repositories].

=StatGen C++ Software=

We have developed a C++ library and tools for handling and analyzing next generation sequencing and genotyping data.

== Library ==

The library contains easy-to-use APIs for developing tools for processing and analyzing next generation sequencing and genotyping data. Allows easy processing of SAM/BAM, GLF, and FASTQ (VCF is coming).

More information on the library can be found at: [[C++ Library: libStatGen]]

The library can be downloaded at: [[libStatGen Download]]

== Programs/Tools ==

Follow the program links for more information on obtaining the tool. Some tools are packaged together.

=== SAM/BAM ===

*[[QPLOT]] - Calculate & plot summary statistics
*[[VerifyBamID]] – Check sample identities for contamination/sample swap
**Genotype concordance based detection
**Estimate based on population allele frequencies without genotype data
*[[Pileup]] – Pileup every base or just bases in specified region and write VCF

==== BAM Util Tools ====
{{BamUtilPrograms}}

=== FASTQ ===
* [[FastQValidator|fastqValidator]] - validate a FASTQ file
**Reports errors for badly formatted files
**Reports Base Composition Statistics (%reads at each read index)

=== Meta Analysis ===
* [[Rare-Metal-Worker|RAREMETALWORKER - generate summary level statistics for meta analysis using Rare-Metal]]
* [[Rare-Metal|RAREMETAL - perform genome-wide meta analysis of rare variants]]

=== Other Tools ===
*[[statgenTools#createUMref|createUMref - Create the University of Michigan formatted reference used by many of our tools]]
*[[Thunder|thunderVCF]]
*[[vcfCooker]] – Manipulate, filter, summarize VCF/BED file in various forms
*[[VcfGenomeStat]] – Print flanking sequences and how often they appear for input VCF file

=== Requested Tools ===

=Other Tools=

* [[samtools-hybrid]] - Since many of our tools still rely on GLF files and samtools stopped supporting GLF files, we created a version of samtools that still supports pileup to GLF files AND incorporates the updated BAQ logic. This version is called samtools-hybrid That code can be downloaded at: https://github.com/statgen/samtools-0.1.7a-hybrid
*[[baseQualityCheck]] - tool to calculate the observed base quality vs. empirical base quality (helps to evaluate mappers)

== Variant Calling ==
* [[glfSingle]] - Variant calling for a single, deeply sequenced individual
* [[glfMultiples]] - Variant calling for multiple, unrelated individuals
* [[Polymutt|polymutt]] - Variant and ''de novo'' mutation detection in families (nuclear or extended pedigrees) from sequencing

== Variant Annotation ==
*[[vcfCodingSnps]] - Annotate coding variants in a VCF file.

== Genotype Imputation ==
*[[Minimac3]] - Fast and Efficient Genotype Imputation.

== Additional Pedigree & Sequence Analysis Tools ==
Can be found at: http://sph.umich.edu/csg/abecasis/software.html

= Other Useful Links =
[[Links to Sequence Analysis Tools]]

= Other =
ASHG 2010 Poster: [[Media:TrostASHG2010.pdf|C++ library & tools for next generation sequence data]]

SAM/BAM Classes FAQs

2021-09-11T21:20:17Z

Hmkang:

[[Category:libStatGen]]
[[Category:libStatGen BAM]]

== How Do I? ==

===== Q: What is SAM/BAM? =====
* A: See: [[SAM]] and [[BAM]]. [[SAM]] also provides additional details as to what is a CIGAR, QUALITY, and TAG.

===== Q: Map between a query/read sequence and a reference sequence to do things like looking up a base/quality at a given reference position or looking up a reference base for a given read position? =====
* A: Use [[C++ Class: CigarRoller#Mapping Between Reference and Read/Query|Cigar::getQueryIndex & Cigar::getRefOffset]]

===== Q: Read only portions of a sorted & indexed BAM file that fall within a specified region? =====
* A:
*# open a sorted and indexed BAM file for reading (<code>SamFile::OpenForRead</code>)
*# read the BAM Index file (<code>SamFile::ReadBamIndex</code>)
*# read the header (<code>SamFile::ReadHeader</code>)
*# set the region to be read (<code>SamFile::SetReadSection</code>)
*# Loop calling <code>SamFile::ReadRecord</code> until it returns false - indicating there are no more reads within this region.
These methods are found in [[C++ Class: SamFile]].

NOTE: ReadRecord will also return false on an error - use <code>SamFile::GetStatus()</code> to determine if it was an error or the end of the region. <code>GetStatus</code> returns SamStatus::NO_MORE_RECS for the end of the region.

See the library usage page for an [[Sam_Library_Usage_Examples#Read_a_Specified_Region_from_a_Sorted_.26_Indexed_BAM_File|Example]].

* Q: How do I read multiple regions?
** A: Use the above solution, but loop on the SetReadSection/ReadRecord steps until all sections have been processed.

===== Q: Get additional information on this library? =====
* A: If you have any questions or comments, please raise issue in [https://github.com/statgen/ our GitHub repositories].

SAM/BAM Classes FAQs

2021-09-11T21:19:55Z

Hmkang:

[[Category:libStatGen]]
[[Category:libStatGen BAM]]

== How Do I? ==

===== Q: What is SAM/BAM? =====
* A: See: [[SAM]] and [[BAM]]. [[SAM]] also provides additional details as to what is a CIGAR, QUALITY, and TAG.

===== Q: Map between a query/read sequence and a reference sequence to do things like looking up a base/quality at a given reference position or looking up a reference base for a given read position? =====
* A: Use [[C++ Class: CigarRoller#Mapping Between Reference and Read/Query|Cigar::getQueryIndex & Cigar::getRefOffset]]

===== Q: Read only portions of a sorted & indexed BAM file that fall within a specified region? =====
* A:
*# open a sorted and indexed BAM file for reading (<code>SamFile::OpenForRead</code>)
*# read the BAM Index file (<code>SamFile::ReadBamIndex</code>)
*# read the header (<code>SamFile::ReadHeader</code>)
*# set the region to be read (<code>SamFile::SetReadSection</code>)
*# Loop calling <code>SamFile::ReadRecord</code> until it returns false - indicating there are no more reads within this region.
These methods are found in [[C++ Class: SamFile]].

NOTE: ReadRecord will also return false on an error - use <code>SamFile::GetStatus()</code> to determine if it was an error or the end of the region. <code>GetStatus</code> returns SamStatus::NO_MORE_RECS for the end of the region.

See the library usage page for an [[Sam_Library_Usage_Examples#Read_a_Specified_Region_from_a_Sorted_.26_Indexed_BAM_File|Example]].

* Q: How do I read multiple regions?
** A: Use the above solution, but loop on the SetReadSection/ReadRecord steps until all sections have been processed.

===== Q: Get additional information on this library? =====
* A: If you have any questions or comments, please raise issue in the corresponding [https://github.com/statgen/ our GitHub repositories].

Software

2021-09-11T21:19:13Z

Hmkang:

[[Category:Software]]
[[Category:C++]]

=Software=
Due to increasing volume of next generation sequencing and genotyping data, we have created these C++ library and tools that use that library.

This page points to downloads, documentation, and papers for software that is written here at the [http://genome.sph.umich.edu Center for Statistical Genetics]

If you have any questions or comments, please raise issue in the corresponding [https://github.com/statgen/ our GitHub repositories].

=StatGen C++ Software=

We have developed a C++ library and tools for handling and analyzing next generation sequencing and genotyping data.

== Library ==

The library contains easy-to-use APIs for developing tools for processing and analyzing next generation sequencing and genotyping data. Allows easy processing of SAM/BAM, GLF, and FASTQ (VCF is coming).

More information on the library can be found at: [[C++ Library: libStatGen]]

The library can be downloaded at: [[libStatGen Download]]

== Programs/Tools ==

Follow the program links for more information on obtaining the tool. Some tools are packaged together.

=== SAM/BAM ===

*[[QPLOT]] - Calculate & plot summary statistics
*[[VerifyBamID]] – Check sample identities for contamination/sample swap
**Genotype concordance based detection
**Estimate based on population allele frequencies without genotype data
*[[Pileup]] – Pileup every base or just bases in specified region and write VCF

==== BAM Util Tools ====
{{BamUtilPrograms}}

=== FASTQ ===
* [[FastQValidator|fastqValidator]] - validate a FASTQ file
**Reports errors for badly formatted files
**Reports Base Composition Statistics (%reads at each read index)

=== Meta Analysis ===
* [[Rare-Metal-Worker|RAREMETALWORKER - generate summary level statistics for meta analysis using Rare-Metal]]
* [[Rare-Metal|RAREMETAL - perform genome-wide meta analysis of rare variants]]

=== Other Tools ===
*[[statgenTools#createUMref|createUMref - Create the University of Michigan formatted reference used by many of our tools]]
*[[Thunder|thunderVCF]]
*[[vcfCooker]] – Manipulate, filter, summarize VCF/BED file in various forms
*[[VcfGenomeStat]] – Print flanking sequences and how often they appear for input VCF file

=== Requested Tools ===

=Other Tools=

* [[samtools-hybrid]] - Since many of our tools still rely on GLF files and samtools stopped supporting GLF files, we created a version of samtools that still supports pileup to GLF files AND incorporates the updated BAQ logic. This version is called samtools-hybrid That code can be downloaded at: https://github.com/statgen/samtools-0.1.7a-hybrid
*[[baseQualityCheck]] - tool to calculate the observed base quality vs. empirical base quality (helps to evaluate mappers)

== Variant Calling ==
* [[glfSingle]] - Variant calling for a single, deeply sequenced individual
* [[glfMultiples]] - Variant calling for multiple, unrelated individuals
* [[Polymutt|polymutt]] - Variant and ''de novo'' mutation detection in families (nuclear or extended pedigrees) from sequencing

== Variant Annotation ==
*[[vcfCodingSnps]] - Annotate coding variants in a VCF file.

== Genotype Imputation ==
*[[Minimac3]] - Fast and Efficient Genotype Imputation.

== Additional Pedigree & Sequence Analysis Tools ==
Can be found at: http://sph.umich.edu/csg/abecasis/software.html

= Other Useful Links =
[[Links to Sequence Analysis Tools]]

= Other =
ASHG 2010 Poster: [[Media:TrostASHG2010.pdf|C++ library & tools for next generation sequence data]]

BamUtil

2021-09-11T21:14:59Z

Hmkang:

[[Category:bamUtil]]
[[Category:C++]]
[[Category:Software]]

= bamUtil Overview =

bamUtil is a repository that contains several programs that perform operations on SAM/BAM files. All of these programs are built into a single executable, <code>bam</code>.

== Getting Help ==

If you have any questions please use the [https://github.com/statgen/bamUtil bamUtil GitHub page] to raise and issue.

See [[BamUtil: FAQ]] to see if your question has already been answered.

== Where to Find It ==
{{ToolGitRepo|repoName=bamUtil}}

== Releases ==

If you prefer to run the last official release rather than the latest development version, you can download that here.

There are two versions of the release, one that include libStatGen and one that does not. If you already have libStatGen installed and want to use your own copy, use the version that does not include libStatGen.

=== Full Release (includes libStatGen) ===

To install an official release, unpack the downloaded file (tar xvf), cd into the bamUtil_x.x.x directory and type make all.

For version 1.0.14 and later, please download libStatGen and bamUtil separately:

'''Version 1.0.14 - Released 7/8/2015'''
*[[LibStatGen Download#Official Releases|libStatGen version 1.0.14]]
*[[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.14]]

'''Older Releases'''
* [[Media:BamUtilLibStatGen.1.0.13.tgz|BamUtilLibStatGen.1.0.13.tgz‎]] - Released 2/20/2015
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.13]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.13]] - see link for version updates

* [[Media:BamUtilLibStatGen.1.0.12.tar.gz|BamUtilLibStatGen.1.0.12.tgz‎]] - Released 5/14/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.12]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.12]] - see link for version updates
** Adds regions to [[BamUtil: mergeBam|mergeBam]]
** Accept ',' delimiters for the tags string input in [[BamUtil: squeeze|squeeze]], [[BamUtil: revert|revert]], & [[BamUtil: diff|diff]]

*[[Media:BamUtilLibStatGen.1.0.11.tar.gz|BamUtilLibStatGen.1.0.11.tar.gz‎]] - Released 2/28/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.11]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.11]] - see link for version updates
** Now properly supports 'B' & 'f' tags
** Cleanup - compile issues

*[[Media:BamUtilLibStatGen.1.0.10.tar.gz|BamUtilLibStatGen.1.0.10.tar.gz‎]] - Released 1/2/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.10]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.10]] - see link for version updates
** Adds PhoneHome/Version checking.

*[[Media:BamUtilLibStatGen.1.0.9.tgz|BamUtilLibStatGen.1.0.9.tgz‎]] - Released 7/7/2013
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.9]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.9]]
** Update to [[BamUtil: mergeBam|mergeBam]]
*** Update to ignore PG lines with duplicate IDs
*** Update to accept merges of matching RG lines
*** Update to log to stderr if no log/out file is specified
* There is no version 1.0.8. It was skipped to stay in line with libStatGen versions (libStatGen 1.0.8 added vcf support)
*[[Media:BamUtilLibStatGen.1.0.7.tgz|BamUtilLibStatGen.1.0.7.tgz‎]] - Released 1/29/2013
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.7]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.7]]
** Update to fix some compile issues on ubuntu 12.10
** Update use of SamRecord::getStringTag to expect the return of a const string pointer due to libStatGen v1.0.7 updates
** Update SamReferenceInfo usage due to libStatGen v1.0.7 updates
** Update to [[BamUtil: diff|diff]]
*** Fix DIFF to test and properly handle running out of available records. Previously no message was printed when this happened and there was a bug for which file it freed
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** Update to facilitate adding other overlap handling functions
** Update to [[BamUtil: mergeBam|mergeBam]] (formerly RGMergeBam)
*** Rename RGMergeBam to MergeBam
*** Update to handle files that already have an RG

*[[Media:BamUtilLibStatGen.1.0.6.tgz|BamUtilLibStatGen.1.0.6.tgz‎]] - Released 11/14/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.6]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.6]]
** Update to [[BamUtil: trimBam|trimBam]]
*** Update to allow trimming a different number of bases from each end of the read
*[[Media:BamUtilLibStatGen.1.0.5.tgz|BamUtilLibStatGen.1.0.5.tgz‎]] - Released 10/24/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.5]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.5]]
** Updates to: [[BamUtil: dedup|dedup]], [[BamUtil: polishBam|polishBam]], [[BamUtil: recab|recab]]
** Update to add compile option to compile without C++0x/C++11
** See [[#Release of just BamUtil (does not include libStatGen)|below]] for additional details on updates
*BamUtilLibStatGen.1.0.4.tgz‎ - Released skipped
*[[Media:BamUtilLibStatGen.1.0.3.tgz|BamUtilLibStatGen.1.0.3.tgz‎]] - Released 09/19/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.3]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.3]]
** Adds: [[BamUtil: dedup|dedup]] [[BamUtil: recab|recab]]
*[[Media:BamUtilLibStatGen.1.0.2.tgz|BamUtilLibStatGen.1.0.2.tgz‎]] - Released 05/16/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.2]]
** Adds: [[BamUtil: bam2FastQ|bam2FastQ]]
*[[Media:BamUtilLibStatGen.1.0.1.tgz|BamUtilLibStatGen.1.0.1.tgz‎]] - Released 05/04/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.1]]
** Adds: [[BamUtil: splitBam|splitBam]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: trimBam|trimBam]], [[BamUtil: polishBam|polishBam]], [[BamUtil: rgMergeBam|rgMergeBam]], [[BamUtil: gapInfo|gapInfo]]
** Adds additional functionality to [[BamUtil: stats|stats]]
** Adds leftShifting to [[BamUtil: writeRegion|writeRegion]] and [[BamUtil: convert|convert]]
** Adds more diff fields to [[BamUtil: diff|diff]]
* [[Media:BamUtilLibStatGen.1.0.0.tgz|BamUtilLibStatGen.1.0.0.tgz‎]] - Released 10/10/2011
**Initial release of bamUtil that includes libStatGen version 1.0.0. It started from the tool found in the deprecated StatGen repository.
**Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.0]] [[BamUtil: validate|validate]], [[BamUtil: convert|convert]], [[BamUtil: dumpHeader|dumpHeader]], [[BamUtil: splitChromosome|splitChromosome]], [[BamUtil: writeRegion|writeRegion]], [[BamUtil: dumpRefInfo|dumpRefInfo]], [[BamUtil: dumpIndex|dumpIndex]], [[BamUtil: readIndexedBam|readIndexedBam]], [[BamUtil: filter|filter]], [[BamUtil: readReference|readReference]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]], [[BamUtil: squeeze|squeeze]], [[BamUtil: findCigars|findCigars]], [[BamUtil: stats|stats]]

=== Release of just BamUtil (does not include libStatGen) ===

To install an official release, unpack the downloaded file (tar xvf), cd into the bamUtil_x.x.x directory and type make all.

'''BamUtil.1.0.14 Release Notes'''
* BamUtil Version 1.0.14 - Released 7/8/2015
** https://github.com/statgen/bamUtil/archive/v1.0.14.tar.gz
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.14]]
** Update [[BamUtil: trimBam|trimBam]]
*** Add option to soft clip (-c) instead of trimming
** Update [[BamUtil: clipOverlap|clipOverlap]]
*** Add option to mark reads as unmapped if they are entirely clipped
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** Add option to gzip the output files
*** Add option to split Read Groups into separate fastq files
*** Add option to get the quality from a tag
** Update [[BamUtil: recab|recab]]
*** Update to ignore ref 'N' when building the recalibration table
*** Add ability to bin
** Add Dedup_LowMem tool

'''Older Releases'''
* BamUtil Version 1.0.13 - Released 2/20/2015
** https://github.com/statgen/bamUtil/archive/v1.0.13.tar.gz
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.13]]
** Makefile Updates
*** Improve logic to determine actual path for the library
*** Update to append to USER_COMPILE_VARS even if specified on the command line
** Update [[BamUtil: writeRegion|writeRegion]]
*** Add option to specify readnames to keep in a file
*** Fixed bug that if a read overlapped 2 BED positions, it was printed twice
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** Update to skip non-primary reads
** Update to [[BamUtil: polishBam|polishBam]]
*** Update to handle '\t' string inputs and to add CO option
*** Fix MD5sum calculation to convert fasta to uppercase prior to calculating

* [[Media:BamUtil.1.0.12.tgz|BamUtil.1.0.12.tgz‎]] - Released 5/14/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.12]]
** Update [[BamUtil: mergeBam|mergeBam]]
*** Add a regions option
** Update to [[BamUtil: squeeze|squeeze]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]]
*** Also accept ',' instead of just ';' as the delimiter in the input tags string.

* [[Media:BamUtil.1.0.11.tgz|BamUtil.1.0.11.tgz‎]] - Released 2/28/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.11]]
*** Adds support for 'B' & 'f' tags that did not work properly before.
** Update [[BamUtil: splitBam|splitBam]] & [[BamUtil: polishBam|polishBam]]
*** Update to work properly if log & output file are not specified (no longer creates '.log')
** Update Main dummy/example tool to indicate the correct tool
** Update to [[BamUtil: bam2FastQ|bam2FastQ]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: filter|filter]], [[BamUtil: mergeBam|mergeBam]], [[BamUtil: splitBam|splitBam]], [[BamUtil: squeeze|squeeze]], [[BamUtil: stats|stats]]
*** Cleanup usage/parameter descriptions
** Update [[BamUtil: revert|revert]]
*** Update compatibility with libStatGen due to 'B' & 'f' tag handling updates
** Add tests for 'B' & 'f' tags

* [[Media:BamUtil.1.0.10.tar.gz|BamUtil.1.0.10.tar.gz‎]] - Released 1/2/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.10]]
** All
*** Add PhoneHome/version checking
*** Make sub-program names case independent
*** Fix Logger.cpp compiler warning
** Adds: [[BamUtil: explainFlags|explainFlags]] - describes the SAM/BAM flags based on the flag value
** Update to [[BamUtil: stats|stats]]
*** Fix Stats to not try to not try to process a record after it is out of the loop (it would already have been processed or is invalid)
** Update to [[BamUtil: splitBam|splitBam]]
*** fix description of --noeof option
** Update to [[BamUtil: writeRegion|writeRegion]]
*** add exclude/required flags
** Update to [[BamUtil: dedup|dedup]] & [[BamUtil: recab|recab]]
*** Ignore secondary reads for dedup and making the recalibration table.
*** skip QC Failures
*** add excludeFlags parameters
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** add exclude flags
*** fix bug for readName sorted when a read is filtered due to flags
*** add sorting validation
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** add --merge option to generate interleaved files.
*** update to open the input file before opening the output files, so if there is an error, the outputs aren't opened
** Update to [[BamUtil: mergeBam|mergeBam]]
*** add option to ignore the RG PI field when checking headers
*** add more informative header merge error messages

* [[Media:BamUtil.1.0.9.tgz|BamUtil.1.0.9.tgz‎]] - Released 7/7/2013
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.9]] (version 1.0.7 should also work)
** Update to [[BamUtil: mergeBam|mergeBam]]
*** Update to ignore PG lines with duplicate IDs
*** Update to accept merges of matching RG lines
*** Update to log to stderr if no log/out file is specified

*[[Media:BamUtil.1.0.7.tgz|BamUtil.1.0.7.tgz‎]] - Released 1/29/2013
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.7]] or above
** Update to fix some compile issues on ubuntu 12.10
** Update use of SamRecord::getStringTag to expect the return of a const string pointer due to libStatGen v1.0.7 updates
** Update SamReferenceInfo usage due to libStatGen v1.0.7 updates
** Update to [[BamUtil: diff|diff]]
*** Fix DIFF to test and properly handle running out of available records. Previously no message was printed when this happened and there was a bug for which file it freed
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** Update to facilitate adding other overlap handling functions
** Update to [[BamUtil: mergeBam|mergeBam]] (formerly RGMergeBam)
*** Rename RGMergeBam to MergeBam
*** Update to handle files that already have an RG
*[[Media:BamUtil.1.0.6.tgz|BamUtil.1.0.6.tgz‎]] - Released 11/14/2012
** Update to [[BamUtil: trimBam|trimBam]]
*** Update to allow trimming a different number of bases from each end of the read
*[[Media:BamUtil.1.0.5.tgz|BamUtil.1.0.5.tgz‎]] - Released 10/24/2012
** Update to [[BamUtil: dedup|dedup]]
*** Update logic for which pair to keep if they have the same quality
** Update to [[BamUtil: polishBam|polishBam]]
*** Update to print the number of successful header additions
** Update to [[BamUtil: recab|recab]]
*** Update to print the number of base skipped due to the base quality
** General Updates
*** Update to add compile option to compile without C++0x/C++11
*BamUtil.1.0.4.tgz‎ - Released skipped
*[[Media:BamUtil.1.0.3.tgz|BamUtil.1.0.3.tgz‎]] - Released 09/19/2012
** Adds: [[BamUtil: dedup|dedup]] [[BamUtil: recab|recab]]
** General Updates
*** Update Logger to write to stderr if output is stdout
** Update to [[BamUtil: stats|stats]]
*** Add required/exclude flags
*** Exclude Clips if excluding umapped
*** Add --withinRegion flag
*** Update phred/qual counts to be uint64_t instead of int to avoid overflow
** Update to [[BamUtil: validate|validate]]
*** Detect header failures
** Update to [[BamUtil: diff|diff]]
*** Update to specify chromosome/pos in ZP as a string rather than int so both can be shown
** Update to [[BamUtil: readReference|readReference]]
*** Output error message if the reference name is not found
** Update to [[BamUtil: splitChromosome|splitChromosome]]
*** Update to actually split the chromosomes and not just hard coded to output chromosomes ids 0-22
** Update Makefile to have cloneLib for cloning libStatGen
*[[Media:BamUtil.1.0.2.tgz|BamUtil.1.0.2.tgz‎]] - Released 05/16/2012
** Adds: [[BamUtil: bam2FastQ|bam2FastQ]]
*[[Media:BamUtil.1.0.1.tgz|BamUtil.1.0.1.tgz‎]] - Released 05/04/2012
** Adds: [[BamUtil: splitBam|splitBam]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: trimBam|trimBam]], [[BamUtil: polishBam|polishBam]], [[BamUtil: rgMergeBam|rgMergeBam]], [[BamUtil: gapInfo|gapInfo]]
** Adds additional functionality to [[BamUtil: stats|stats]]
** Adds leftShifting to [[BamUtil: writeRegion|writeRegion]] and [[BamUtil: convert|convert]]
** Adds more diff fields to [[BamUtil: diff|diff]]
*[[Media:BamUtil.1.0.0.tgz|BamUtil.1.0.0.tgz‎]] - Released 10/10/2011
**Initial release of just bamUtil. It started from the tool found in the deprecated StatGen repository.
**Contains: [[BamUtil: validate|validate]], [[BamUtil: convert|convert]], [[BamUtil: dumpHeader|dumpHeader]], [[BamUtil: splitChromosome|splitChromosome]], [[BamUtil: writeRegion|writeRegion]], [[BamUtil: dumpRefInfo|dumpRefInfo]], [[BamUtil: dumpIndex|dumpIndex]], [[BamUtil: readIndexedBam|readIndexedBam]], [[BamUtil: filter|filter]], [[BamUtil: readReference|readReference]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]], [[BamUtil: squeeze|squeeze]], [[BamUtil: findCigars|findCigars]], [[BamUtil: stats|stats]]

== Citation ==
If you use BamUtil, please cite our publication on GotCloud which includes BamUtil:
[http://genome.cshlp.org/content/early/2015/04/14/gr.176552.114.abstract Jun, Goo, et al. "An efficient and scalable analysis framework for variant extraction and refinement from population scale DNA sequence data." Genome research (2015): gr-176552.]

= Programs =

The software reads the beginning of an input file to determine if it is SAM/BAM. To determine the format (SAM/BAM) of the output file, the software checks the output file's extension. If the extension is ".bam" it writes a BAM file, otherwise it writes a SAM file.

{{BamUtilPrograms}}

EPACTS

2019-02-19T12:03:06Z

Hmkang: /* Output Files */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* Dec 15th, 2016 : EPACTS v3.3.0 release (github)
** Moved the repository into github
** Some major fixes in handling large sample size (>18,000)
** Other minor bug fixes
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at https://github.com/statgen/EPACTS
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.spa2
| Binary
| YES <br>
| Moderate
| Saddlepoint Approximation Method
| Shawn Lee & Rounak Dey
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

$ git clone https://github.com/statgen/EPACTS.git
$ cd EPACTS
$ ./configure --prefix [/path/to/install]
$ make
$ make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts/

* If you want to access previous versions, visit http://csg-old.sph.umich.edu/kang/epacts/download

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

Minimac Diagnostics

2017-06-08T15:35:20Z

Hmkang: /* Estimated Imputation Accuracy */

'''[[minimac]]''' is a tool for imputation of missing genotypes into phased haplotypes. At the end of each run, [[minimac]] generates summaries of imputation quality and stores those in a ''.info'' file.

== Basic Descriptors ==

=== Marker and Allele Labels ===

The first three columns in the ''.info'' file list marker name and alleles for each marker. Typically, the most common allele will be listed first, but this is not guaranteed.

=== Estimated Allele Frequency ===

The next column in the ''.info'' file lists the estimated frequency of allele 1 - this corresponds to the average number of imputed copies of allele 1 for each individual, divided by two.

=== Estimated Imputation Accuracy (ImpRsq) ===

Frequency information is followed by an estimate of the squared correlation between imputed genotypes and true, unobserved genotypes. Since true genotypes are not available, this calculation is based on the idea that poorly imputed genotype counts will shrink towards their expectations based on population allele frequencies alone; specifically <math>2p</math> where <math>p</math> is the frequency of the allele being imputed.

Currently, minimac uses the following definition:

:<math>\hat{r}^2 = {{Var(\mbox{Estimated Counts})}\over{\hat{p}(1-\hat{p})}}</math>

== Leave One Out Statistics ==

To evaluate imputation quality, Minimac hides data for each genotyped SNP in turn and calculates 3 statistics, described below.

=== looRsq : Estimated R-squared in Leave-One-Out Analysis ===

This first statistic is calculated by hiding all known genotypes for the SNP, imputing it and then estimating imputation accuracy. It doesn't use the known genotypes for the SNP at all.

=== empR : Correlation Between Imputed and True Genotypes ===

Whereas looRsq statistic completely ignores experimental genotypes, this one is based on a comparison of imputed and experimental genotypes. A negative correlation between imputed and experimental genotypes can indicate allele flips.

=== empRsq : Squared Correlation Between Imputed and True Genotypes ===

Whereas looRsq statistic reports the estimated imputation accuracy, this one reports the ''actual'' imputation accuracy - as estimated by comparing genotypes generated using imputation (after hiding any known genotypes for the marker) and the previously hidden known genotypes. By comparing empRsq and looRsq it should be possible to tell whether estimates of imputation accuracy are well calibrated.

EPACTS

2017-06-01T19:24:40Z

Hmkang: /* Single Variant Tests */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* Dec 15th, 2016 : EPACTS v3.3.0 release (github)
** Moved the repository into github
** Some major fixes in handling large sample size (>18,000)
** Other minor bug fixes
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at https://github.com/statgen/EPACTS
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.spa2
| Binary
| YES <br>
| Moderate
| Saddlepoint Approximation Method
| Shawn Lee & Rounak Dey
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

$ git clone https://github.com/statgen/EPACTS.git
$ cd EPACTS
$ ./configure --prefix [/path/to/install]
$ make
$ make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts/

* If you want to access previous versions, visit http://csg-old.sph.umich.edu/kang/epacts/download

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/epacts/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

EPACTS

2017-02-28T11:50:11Z

Hmkang: /* Currently Supported Statistical Tests */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* Dec 15th, 2016 : EPACTS v3.3.0 release (github)
** Moved the repository into github
** Some major fixes in handling large sample size (>18,000)
** Other minor bug fixes
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at https://github.com/statgen/EPACTS
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.spa
| Binary
| YES <br>
| Moderate
| Saddlepoint Approximation Method
| Shawn Lee & Rounak Dey
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

$ git clone https://github.com/statgen/EPACTS.git
$ cd EPACTS
$ ./configure --prefix [/path/to/install]
$ make
$ make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts/

* If you want to access previous versions, visit http://csg-old.sph.umich.edu/kang/epacts/download

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/epacts/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

EPACTS

2016-12-15T22:50:03Z

Hmkang: /* Installation Details */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* Dec 15th, 2016 : EPACTS v3.3.0 release (github)
** Moved the repository into github
** Some major fixes in handling large sample size (>18,000)
** Other minor bug fixes
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at https://github.com/statgen/EPACTS
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.spa
| Binary
| YES <br>
| Moderate
| Saddlepoint Approximation Method
| Shawn Lee & Rounak Dey
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.score
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Quantitative Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

$ git clone https://github.com/statgen/EPACTS.git
$ cd EPACTS
$ ./configure --prefix [/path/to/install]
$ make
$ make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts/

* If you want to access previous versions, visit http://csg-old.sph.umich.edu/kang/epacts/download

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/epacts/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

EPACTS

2016-12-15T22:48:50Z

Hmkang: /* Getting Started With Examples */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* Dec 15th, 2016 : EPACTS v3.3.0 release (github)
** Moved the repository into github
** Some major fixes in handling large sample size (>18,000)
** Other minor bug fixes
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at https://github.com/statgen/EPACTS
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.spa
| Binary
| YES <br>
| Moderate
| Saddlepoint Approximation Method
| Shawn Lee & Rounak Dey
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.score
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Quantitative Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

$ git clone https://github.com/statgen/EPACTS.git
$ cd EPACTS
$ ./configure --prefix [/path/to/install]
$ make
$ make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/bin/epacts/

* If you want to access previous versions, visit http://csg-old.sph.umich.edu/kang/epacts/download

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/tools/epacts-3.3.0/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/epacts/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

EPACTS

2016-12-15T22:47:50Z

Hmkang: /* Obtaining EPACTS */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* Dec 15th, 2016 : EPACTS v3.3.0 release (github)
** Moved the repository into github
** Some major fixes in handling large sample size (>18,000)
** Other minor bug fixes
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at https://github.com/statgen/EPACTS
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.spa
| Binary
| YES <br>
| Moderate
| Saddlepoint Approximation Method
| Shawn Lee & Rounak Dey
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.score
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Quantitative Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

$ git clone https://github.com/statgen/EPACTS.git
$ cd EPACTS
$ ./configure --prefix [/path/to/install]
$ make
$ make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/bin/epacts/

* If you want to access previous versions, visit http://csg-old.sph.umich.edu/kang/epacts/download

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/epacts/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

EPACTS

2016-12-15T22:47:16Z

Hmkang: /* Installation Details */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* Dec 15th, 2016 : EPACTS v3.3.0 release (github)
** Moved the repository into github
** Some major fixes in handling large sample size (>18,000)
** Other minor bug fixes
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at http://csg-old.sph.umich.edu/kang/epacts/download/index.html
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.spa
| Binary
| YES <br>
| Moderate
| Saddlepoint Approximation Method
| Shawn Lee & Rounak Dey
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.score
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Quantitative Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

$ git clone https://github.com/statgen/EPACTS.git
$ cd EPACTS
$ ./configure --prefix [/path/to/install]
$ make
$ make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/bin/epacts/

* If you want to access previous versions, visit http://csg-old.sph.umich.edu/kang/epacts/download

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/epacts/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

EPACTS

2016-12-15T22:45:32Z

Hmkang: /* Single Variant Tests */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* Dec 15th, 2016 : EPACTS v3.3.0 release (github)
** Moved the repository into github
** Some major fixes in handling large sample size (>18,000)
** Other minor bug fixes
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at http://csg-old.sph.umich.edu/kang/epacts/download/index.html
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.spa
| Binary
| YES <br>
| Moderate
| Saddlepoint Approximation Method
| Shawn Lee & Rounak Dey
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.score
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Quantitative Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

*Download EPACTS source distribution at http://csg-old.sph.umich.edu/kang/epacts/download/EPACTS-3.2.6.tar.gz (165MB)
*Uncompress EPACTS package, and install the package using the following set of commands
* Visit http://csg-old.sph.umich.edu/kang/epacts/download for checking previous versions

tar xzvf EPACTS-3.2.6.tar.gz
cd EPACTS-3.2.6
./configure --prefix=/path/to/install
make
make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/bin/epacts/

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/epacts/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

EPACTS

2016-12-15T22:44:26Z

Hmkang: /* Lastest ChangeLog */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* Dec 15th, 2016 : EPACTS v3.3.0 release (github)
** Moved the repository into github
** Some major fixes in handling large sample size (>18,000)
** Other minor bug fixes
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at http://csg-old.sph.umich.edu/kang/epacts/download/index.html
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.score
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Quantitative Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

*Download EPACTS source distribution at http://csg-old.sph.umich.edu/kang/epacts/download/EPACTS-3.2.6.tar.gz (165MB)
*Uncompress EPACTS package, and install the package using the following set of commands
* Visit http://csg-old.sph.umich.edu/kang/epacts/download for checking previous versions

tar xzvf EPACTS-3.2.6.tar.gz
cd EPACTS-3.2.6
./configure --prefix=/path/to/install
make
make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/bin/epacts/

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/epacts/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

EPACTS

2016-05-01T17:22:24Z

Hmkang: /* Single Variant Tests */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at http://www.sph.umich.edu/csg/kang/epacts/ .
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.score
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Quantitative Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

*Download EPACTS source distribution at http://www.sph.umich.edu/csg/kang/epacts/download/EPACTS-3.2.6.tar.gz (165MB)
*Uncompress EPACTS package, and install the package using the following set of commands
* Visit http://www.sph.umich.edu/csg/kang/epacts/download for checking previous versions

tar xzvf EPACTS-3.2.6.tar.gz
cd EPACTS-3.2.6
./configure --prefix=/path/to/install
make
make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/bin/epacts/

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/epacts/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

EPACTS

2016-05-01T17:21:57Z

Hmkang: /* Single Variant Tests */

'''EPACTS''' (Efficient and Parallelizable Association Container Toolbox) is a versatile software pipeline to perform various statistical tests for identifying genome-wide association from sequence data through a user-friendly interface, both to scientific analysts and to method developers.

== Join in EPACTS mailing list ==

Please join in the [http://groups.google.com/group/epacts EPACTS Google Group] to ask / discuss / comment about EPACTS.

== Lastest ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs (more comprehensive fix is still pending)
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
See [[#Full ChangeLog]] for full details

== Key Features ==

EPACTS currently provides the following set of key features
* Robust support for widely used format of sequence-based genotypes (VCF) and phenotypes with pedigree (PED)
** Efficient library for accessing VCF file to reduce computational burden to analyze large-scale sequencing data
** Support selecting markers by arbitrary combination of substring matching.
** Support for using genotype dosages instead of hard genotype calls
** Utilize PED format to perform test across multiple traits.
* Supports a large number of widely used statistical tests for single variant association and burden tests.
** See the "Currently Supported Statistical Tests" section below for more information
* Easy to Highly Parallelize Jobs
** Makefile-based partition into and ligation of multiple subtasks
** Parallel run of job is simply adding one parameter when running EPACTS
* Integrative and versatile framework that allows easy addition of additional statistical test
** Core input/output routines are implemented in C++
** Most statistical tests (except for EMMAX) are implemented in R
** Adding a simple R function to implement additional statistical test (See [[#Implementing Additional Statistical Tests]] for details)
* Useful utilities for post-association-analysis tasks
** Automatic functional annotation of associated variants
** Automatic generation of QQ and Manhattan Plot
** (TBA) Zoom plot for the significant associations

== Obtaining EPACTS ==

* The official release of EPACTS software is available at http://www.sph.umich.edu/csg/kang/epacts/ .
** From the CSG cluster, it is available at /net/fantasia/home/bin/epacts/
* Note that R (version 2.10 or higher) and gnuplot (version 4.2 or higher) must be installed in order to run EPACTS correctly.

== Currently Supported Statistical Tests ==

EPACTS supports the following sets of widely used statistical tests for single variant tests and burden tests

=== Single Variant Tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.wald
| Binary
| YES <br> (Joint)
| Slow
| Logisitic Wald Test
| Hyun Min Kang <br> (simply used glm in R)
|-
| b.score
| Binary
| YES <br> (Regressed Out)
| Fast
| Logistic Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma & Hyun Min Kang
|-
| b.firth
| Binary
| YES <br> (Joint)
| Slow
| Firth Bias-Corrected Logistic Likelihood Ratio Test
| Clement Ma
|-
| b.lrt
| Binary
| YES <br> (Joint)
| Slow
| Likelihood Ratio Test
| Clement Ma
|-
| b.glrt
| Binary
| NO
| Fast
| Genotype Likelihood Ratio Test <br> (use GL or PL field in VCF to perform case-control test)
| Hyun Min Kang
|-
| q.lm
| Quantitative
| YES <br> (Joint)
| Slow
| Linear Wald Test
| Hyun Min Kang <br> (as implemented in lm in R)
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Quantitative Score Test <br> (from Lin DY and Tang ZZ, AJHG 2011 89:354-67)
| Clement Ma
|-
| q.linear
| Quantitative
| YES <br> (Regressed Out)
| Fast
| Linear Wald Test
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint)
| Slow
| Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression <br> of phenotypes on binary genotypes (dominant model)
| Hyun Min Kang
|-
| q.emmax
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| EMMAX <br> ( Kang et al (2010) Nat Genet 42:348-54 )
| Hyun Min Kang
|-
|}

=== Gene-wise or group-wise tests ===

<noinclude>
{|</noinclude> border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse; font-size: 95%; clear: center;"<noinclude>
|-
! Test Name
! Phenotypes
! Covariates
! Computational Time
! Description
| Implemented by
|-
| b.collapse
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and 0/1 collapsed variables
| Hyun Min Kang
|-
| b.madsen
| Binary
| NO
| Slow
| Wilcoxon Rank Sum Test between binary phenotypes and weighted rare variant scores (slightly different version from the published method - it uses pooled allele frequency across cases and controls for weighting each variant)
| Hyun Min Kang
|-
| b.wcnt
| Binary
| YES <br> (Joint Estimation)
| Slow
| Logistic Wald Test between binary phenotypes and weighted rare variant scores
| Hyun Min Kang
|-
| q.reverse
| Quantitative
| YES <br> (Joint Estimation)
| Slow
| Reverse regression of phenotypes on binary collapsed variables
| Hyun Min Kang
|-
| q.wilcox
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| Nonparametric Reverse regression of phenotypes on collapsed variables
| Hyun Min Kang
|-
| skat
| Binary/Quantitative
| YES <br> (Joint Estimation)
| Slow
| SKAT-O Test by Lee et al, Biostatistics (2012)
| Seunggeun Lee <br> (adaptive by Xueling Sim and Hyun Min Kang)
|-
| VT
| Binary/Quantitative
| YES <br> (Regressed out first)
| Slow
| Variable Threshold Test <br> with adaptive permutation <br> Price et al, AJHG (2010) 86:832-8
| Hyun Min Kang
|-
| emmaxCMC
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Collapsing burden test using EMMAX
| Hyun Min Kang
|-
| emmaxVT
| Binary/Quantitative
| YES <br> (Regressed Out First)
| Slow
| Variable-threshold burden test using EMMAX
| Hyun Min Kang
|-
| mmskat
| Quantitative
| YES <br> (Regressed Out First)
| Slow
| SKAT test using EMMAX
| Seunggeun Lee & Hyun Min Kang
|}

== Installation Details ==

If you want to use EPACTS in an Ubuntu platform, following the step below

*Download EPACTS source distribution at http://www.sph.umich.edu/csg/kang/epacts/download/EPACTS-3.2.6.tar.gz (165MB)
*Uncompress EPACTS package, and install the package using the following set of commands
* Visit http://www.sph.umich.edu/csg/kang/epacts/download for checking previous versions

tar xzvf EPACTS-3.2.6.tar.gz
cd EPACTS-3.2.6
./configure --prefix=/path/to/install
make
make install

(Important Note: '''make sure to specify --prefix=/path/to/install''' to avoid installing to the default path /usr/local/, which you may not have the permission. /home/your_userid/epacts might be a good one, if you are not sure where to install)

* Now ${EPACTS_DIR} represents the '/path/to/install' directory

* Download the reference FASTA files from 1000 Genomes FTP automatically by running the following commands

${EPACTS_DIR}/bin/epacts download

(For advanced users, to save time for downloading the FASTA files (~900MB), you may copy a local copy of GRCh37 FASTA file and the index file to ${EPACTS_DIR}/share/EPACTS/)

*Perform a test run by running the following command

${EPACTS_DIR}/bin/test_run_epacts.sh

In order to use EPACTS in the CSG cluster, you do not need to install them. You can directly use or make a copy of the in-house release version at

/net/fantasia/home/hmkang/bin/epacts/

== Getting Started With Examples ==
If you are using EPACTS from the CSG cluster, please set the following environment variable
EPACTS_DIR=/net/fantasia/home/hmkang/bin/epacts (in bash)
setenv EPACTS_DIR /net/fantasia/home/hmkang/bin/epacts (in csh)

If you downloaded EPACTS binary and please set EPACTS_DIR to the full path of the downloaded and uncompressed directory.

=== All-in-one example ===

To get started with EPACTS, run the following command will perform an example run
${EPACTS_DIR}/bin/test_run_epacts.sh

You will find a series of lines in test_run_epacts.sh script commented out for each possible test.

The example phenotype (PED format) and genotype (VCF format) can be found at
${EPACTS_DIR}/share/EPACTS/

=== Single Variant Test ===

Or You can run EPACTS command yourself by running
${EPACTS_DIR}/epacts single \
--vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped \
--min-maf 0.001 --chr 20 --pheno DISEASE --cov AGE --cov SEX --test b.score --anno \
--out out/test --run 2

The command above will perform single variant association test using a dummy case-control phenotype file and a subset of 1000 genomes exome VCF file (chr20) using score test statistic for all variants over 1% of higher MAF using 2 parallel runs.

You will see the 4 output files as the main outcome of the analysis

==== Output Text of All Test Statistics ====

The filename is out/test.single.b.score.epacts.gz and the contents will look like
$ zcat out/test.single.b.score.epacts.gz | head
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 68303 68303 20:68303_A/G_Upstream:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 68319 68319 20:68319_C/A_Upstream:DEFB125 266 1.4467e-36 1 0 NA NA NA NA NA NA
20 68396 68396 20:68396_C/T_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76635 76635 20:76635_A/T_Intron:DEFB125 266 1.534e-37 1 0 NA NA NA NA NA NA
20 76689 76689 20:76689_T/C_Synonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76690 76690 20:76690_T/C_Nonsynonymous:DEFB125 266 1 1 0.0018797 NA NA NA NA NA NA
20 76700 76700 20:76700_G/A_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76726 76726 20:76726_C/G_Nonsynonymous:DEFB125 266 0 1 0 NA NA NA NA NA NA
20 76771 76771 20:76771_C/T_Nonsynonymous:DEFB125 266 3 1 0.0056391 0.68484 0.40587 145 121 0.013793 0.0082645

==== Output Text of Top Associations ====

Same type of file but containing top 5,000 association will be stored at out/test.epacts.top5000

$ head out/test.single.b.score.epacts.top5000
#CHROM BEGIN END MARKER_ID NS AC CALLRATE MAF PVALUE SCORE N.CASE N.CTRL AF.CASE AF.CTRL
20 1610894 1610894 20:1610894_G/A_Synonymous:SIRPG 266 138.64 1 0.26061 6.9939e-05 3.9765 145 121 0.65177 0.36476
20 4162411 4162411 20:4162411_T/C_Intron:SMOX 266 204 1 0.38346 0.00055583 -3.4523 145 121 0.62759 0.93388
20 34061918 34061918 20:34061918_T/C_Intron:CEP250 266 41.815 1 0.0786 0.00095471 3.3035 145 121 0.22543 0.075436
20 4155948 4155948 20:4155948_G/A_Intron:SMOX 266 215 1 0.40414 0.0020792 -3.0787 145 121 0.68276 0.95868
20 4680251 4680251 20:4680251_A/G_Nonsynonymous:PRNP 266 186 1 0.34962 0.0025962 3.0119 145 121 0.8069 0.57025
20 36668874 36668874 20:36668874_G/A_Synonymous:RPRD1B 266 96 1 0.18045 0.003031 2.9646 145 121 0.44828 0.2562
20 36641871 36641871 20:36641871_G/A_Synonymous:TTI1 266 10 1 0.018797 0.004308 -2.8547 145 121 0.0068966 0.07438
20 1616892 1616892 20:1616892_A/G_Synonymous:SIRPG 266 144 1 0.27068 0.0051239 2.7991 145 121 0.63449 0.42975
20 25038372 25038372 20:25038372_G/A_Intron:ACSS1 266 103.3 1 0.19418 0.005748 2.7618 145 121 0.47201 0.28813

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''AC''' : Total Non-reference Allele Count
* '''CALLRATE''' : Fraction of non-missing genotypes.
* '''MAF''' : Minor allele frequencies
* '''PVALUE''' : P-value of single variant test
* '''AF.CASE''' : Non-reference allele frequencies for cases
* '''AF.CTRL''' : Non-reference allele frequencies for controls

==== Q-Q plot of test statistics (stratified by MAF) ====

The file out/test.b.score.epacts.qq.pdf will be generated as shown below

[[File:test_b_score_epacts_qq.png]]

==== Manhattan Plot of Test Statistics ====

The file out/test.b.score.epacts.mh.pdf will be generated for chr20 only.

[[File:test_b_score_epacts_mh.png]]

An example Genome-wide manhattan plot (from a genome-wide run) will look like below

[[File:tes_b_score_epacts_mh_gw.png]]

=== Gene-wise or group-wise burden test ===

Gene-wise or group-wise burden test requires two steps. First, 'group' file containing the list of
markers per group needs to be generated. Second, group-wise burden test needs to be run

==== Creating marker group file ====

The marker group file has the following format

[GROUP_ID] [MARKER_ID_1] [MARKER_ID_2] .... [MARKER_ID_N]

where
* [GROUP_ID] is a string representing the group (e.g. gene name)
* [MARKER_ID_K] is a marker key as a format of [CHROM]:[POS]_[REF]/[ALT] (NOTE THAT THIS IS DIFFERENT FROM TYPICAL VCF MARKER ID field)

Note that [MARKER_ID_K] has to be sorted by increasing order of genomic coordinate

In order to create gene-level group file from typically formatted VCF file, one may use the following utility

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --nonsyn

The above command create a file [output-group-file] containing a list of missense and nonsense variants per each gene. To incorporate different types of functional annotations, use --type option as follows

${EPACTS_DIR}/epacts make-group --vcf [input-vcf] --out [output-group-file] --format [epacts, annovar, chaos or gatk] --type [function_type_1] --type [function_type_2] ...

Type 'epacts makegroup -man' for the detailed documentation

==== Annotating VCF file using EPACTS ====

If the VCF is not annotated, 'epacts makegroup' cannot be used. In order to annotate VCF, one can use the example VCF using ANNOVAR as follows:

${EPACTS_DIR}/epacts anno \
--in ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.vcf.gz \
--out ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz

The epacts anno script will add "ANNO=[function]:[genename]" entry into the INFO field based on gencodeV7 (default) or refGene database.

It is important to check whether the VCF file is already annotated or not in order to avoid no or redundant annotation.

==== Running Groupwise Test ====

To perform a groupwise burden test on the example VCF (annotated as above), run the following command

${EPACTS_DIR}/epacts group --vcf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.vcf.gz \
--groupf ${EPACTS_DIR}/data/1000G_exome_chr20_example_softFiltered.calls.anno.grp --out out/test.gene.skat \
--ped ${EPACTS_DIR}/data/1000G_dummy_pheno.ped --maxAF 0.05 \
--chr 20 --pheno QT --cov AGE --cov SEX --test skat --skat-o --run 2

==== Example Output ====
$ head out/test.gene.skat.epacts.top5000
#CHROM BEGIN END MARKER_ID NS FRAC_WITH_RARE NUM_ALL_VARS NUM_PASS_VARS NUM_SING_VARS PVALUE STATRHO
20 62607037 62608720 20:62607037-62608720_SAMD10 266 0.14662 9 5 1 0.0020064 1
20 2816211 2820493 20:2816211-2820493_FAM113A 266 0.011278 12 2 1 0.0032542 0
20 47245987 47361692 20:47245987-47361692_PREX1 266 0.1391 54 9 6 0.0054849 1
20 34761734 34810279 20:34761734-34810279_EPB41L1 266 0.071429 14 7 5 0.0068492 0.2
20 61340671 61391602 20:61340671-61391602_NTSR1 266 0.11278 24 9 3 0.011063 1
20 48561952 48568644 20:48561952-48568644_RNF114 266 0.011278 4 2 1 0.015175 0.2
20 60962895 60963559 20:60962895-60963559_RPS21 266 0.06015 6 3 2 0.016409 0
20 55904961 55917801 20:55904961-55917801_SPO11 266 0.011278 11 3 3 0.018031 0

The key columns represents:
* '''NS''' : Number of phenotyped samples with non-missing genotypes
* '''FRAC_WITH_RARE''' : Fraction of individual carrying rare variants below --max-maf (default : 0.05) threshold.
* '''NUM_ALL_VARS''' : Number of all variants defining the group.
* '''NUM_PASS_VARS''' : Number of variants passing the --min-maf, --min-mac, --max-maf, --min-callrate thresholds
* '''NUM_SING_VARS''' : Number of singletons among variants in NUM_PASS_VARS
* '''PVALUE''' : P-value of burden tests
* Other columns are test specific auxiliary columns. For example, in the VT test, the optimal MAF threshold is recorded as an auxiliary output column.

=== Specialized Instruction for EMMAX tests ===

EMMAX (Efficient Mixed Model Association eXpedited - Kang et al (2010) Nat Genet 42:348-54) is an efficient implementation of mixed model association accounting for sample structure including population structure and hidden relatedness. Currently EPACTS supports EMMAX association mapping in single variant test and CMC-like burden tests.

Because EMMAX is based on linear model, the method fits better to quantiative traits than binary traits. However, p-values for binary traits are expected to be valid in the spirit of Armitage trend test, although the estimated effect size may not be precise.

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

==== Single Variant EMMAX Association Analysis ====

* '''Creating Kinship Matrix''' : From VCF, we recommend to set a MAF (e.g. 0.01) and call rate (e.g. 0.95) threshold to select high-quality markers to generate kinship matrix as follows.
${EPACTS_DIR}/epacts make-kin \
--vcf [input.vcf.gz] --ped [input.ped (Optional)] --min-maf 0.01 --minCallRate 0.95 \
--sepchr (if VCF is separated by chromosome) --out [outprefix.kinf] --run [# of parallel jobs]

If you provide [input.ped] file, then it will calculate the subset the individuals contained in the PED file.

The procedure above will create a file [outprefix.kinf] after splitting and merging the genomes into multiple pieces. If only a certain subset of SNPs needs to be considered due to target regions, LD-pruning, or any other reasons, a VCF containing the subset of markers must be created beforehand and should be used as input VCF file.

* '''Perform Single Variant Association''' : From VCF and PED, we recommend to use less stringent MAF threshold (e.g. 0.001) and call rate (e.g. 0.50) to perform single variant association
${EPACTS_DIR}/epacts single \
--vcf [input.vcf.gz] --ped [input.ped] --min-maf 0.001 --kin [outputprefix.kinf] \
--sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] --test q.emmax \
--out [outprefix] --run [# of parallel jobs]

The procedure above will perform single variant association analysis compatible to other types of single variant association analyses implemented in EPACTS

==== Burden-style EMMAX Association Analysis ====

In order to run EMMAX analysis from sequence-based genotypes. We recommend running EPACTS multiple times using the following procedure.

* '''Creating Kinship Matrix''' : See 'Creating Kinship Matrix' section in [[#Single Variant EMMAX Association Analysis]]
* '''Create Marker Group'''
** By annotating the VCF and extracting missense and nonsense variants
*** [[#Annotating VCF file using ANNOVAR]] - This step will be required to create marker group file
*** [[#Creating marker group file]] - Assume that [group.grp] file is produced
** Or, by creating your own marker group information
*** See [[#Creating marker group file]] for details
* Run CMC-style burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxCMC --out [outprefix]
* Run Variable Threshold burden test by
${EPACTS_DIR}/epacts group --groupf [group.grp] \
--vcf [input.vcf.gz] --ped [input.ped] --max-maf [max-MAF-for-rare-variants] \
--kin [outputprefix.kinf] --sepchr --pheno [PHENO_NAME] --cov [COV1] --cov [COV2] \
--test emmaxVT --out [outprefix]

== Preparing Your Own Input Data ==

=== VCF file for Genotypes ===

EPACTS support VCF files as input for association with the following requirement
* Input VCF file must be bgzipped and tabixed before running association to allow efficient random access of the file. Below is an example command to conver plain VCF into bgzipped and tabixed VCF
bgzip input.vcf ## this command will produce input.vcf.gz
tabix -pvcf -f input.vcf.gz ## this command will produce input.vcf.gz.tbi
* If the VCF file is separated by chromosome, the VCF file specified in the input argument must contain the string "chr1" in the chromosome 1 file, and corresponding chromosome name for other chromosomes. Thus, the files names should be like <code>[prefix]chr1[suffix].vcf.gz</code>, <code>[prefix]chr2[suffix].vcf.gz</code>, ..., <code>[prefix]chr22[suffix].vcf.gz</code>, <code>[prefix]chrX[suffix].vcf.gz</code>.
* Sample IDs in the VCF file must be consistent to those from PED file
* Currently EPACTS only support bi-allelic variants, but it handles SNPs, INDELs, snd SVs.
* Currently, EPACTS only support VCF aligned with NCBI build 37 coordinates
* An example VCF file from 1000 genome project is below.
$ zcat example/1000G_integrated_phase1_chr20.vcf.gz | cut -f 1-10 | head -50
##fileformat=VCFv4.1
##INFO=<ID=LCSNP,Number=0,Type=Flag,Description="Genotype likelihood in Low coverage VCF in data integration">
##INFO=<ID=EXSNP,Number=0,Type=Flag,Description="Genotype likelihood in Exome VCF in data integration">
##INFO=<ID=INDEL,Number=0,Type=Flag,Description="Genotype likelihood in INDEL VCF in data integration">
##INFO=<ID=SV,Number=0,Type=Flag,Description="Genotype likelihood in SV VCF in data integration">
##INFO=<ID=BAVGPOST,Number=1,Type=Float,Description="Average posterior probability from beagle">
##INFO=<ID=BRSQ,Number=1,Type=Float,Description="Genotype imputation quality estimate from beagle">
##INFO=<ID=LDAF,Number=1,Type=Float,Description="MLE Allele Frequency Accounting for LD">
##INFO=<ID=AVGPOST,Number=1,Type=Float,Description="Average posterior probability from MaCH/Thunder">
##INFO=<ID=RSQ,Number=1,Type=Float,Description="Genotype imputation quality from MaCH/Thunder">
##INFO=<ID=ERATE,Number=1,Type=Float,Description="Per-marker Mutation rate from MaCH/Thunder">
##INFO=<ID=THETA,Number=1,Type=Float,Description="Per-marker Transition rate from MaCH/Thunder">
##INFO=<ID=CIEND,Number=2,Type=Integer,Description="Confidence interval around END for imprecise variants">
##INFO=<ID=CIPOS,Number=2,Type=Integer,Description="Confidence interval around POS for imprecise variants">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##INFO=<ID=HOMLEN,Number=.,Type=Integer,Description="Length of base pair identical micro-homology at event breakpoints">
##INFO=<ID=HOMSEQ,Number=.,Type=String,Description="Sequence of base pair identical micro-homology at event breakpoints">
##INFO=<ID=SOURCE,Number=.,Type=String,Description="Source of deletion call">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=AC,Number=.,Type=Integer,Description="Alternate Allele Count">
##INFO=<ID=AN,Number=1,Type=Integer,Description="Total Allele Count">
##ALT=<ID=DEL,Description="Deletion">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=DS,Number=1,Type=Float,Description="Genotype dosage from MaCH/Thunder">
##FORMAT=<ID=GL,Number=.,Type=Float,Description="Genotype Likelihoods">
##FORMAT=<ID=BD,Number=1,Type=Float,Description="Genotype dosage from beagle">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT HG00096
20 60479 . C T 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.894;LDAF=0.0020;AVGPOST=0.9995;RSQ=0.8779;ERATE=0.0005;THETA=0.0008;AC=4;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.19,-0.46,-2.68:0.0022
20 60522 . T TC 1588 PASS INDEL;BAVGPOST=1.000;BRSQ=0.994;LDAF=0.0116;AVGPOST=0.9980;RSQ=0.9327;ERATE=0.0004;THETA=0.0167;AC=24;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-0.90,-9.20:0
20 60571 . C A 100 PASS LCSNP;EXSNP;BAVGPOST=0.999;BRSQ=0.813;LDAF=0.0029;AVGPOST=0.9986;RSQ=0.8085;ERATE=0.0014;THETA=0.0014;AC=5;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.05,-0.96,-5.00:0.0008
20 60795 . G C 100 PASS LCSNP;EXSNP;BAVGPOST=1.000;BRSQ=0.930;LDAF=0.0006;AVGPOST=0.9996;RSQ=0.7205;ERATE=0.0003;THETA=0.0041;AC=1;AN=2184 GT:DS:GL:BD 0|0:0.000:-0.03,-1.21,-5.00:0.0001
20 60810 . G GA 127 PASS INDEL;BAVGPOST=1.000;BRSQ=0.862;LDAF=0.0013;AVGPOST=0.9987;RSQ=0.5684;ERATE=0.0004;THETA=0.0061;AC=2;AN=2184 GT:DS:GL:BD 0|0:0.000:0.00,-1.80,-18.80:0

=== PED file for Phenotypes and Covariates ===

EPACTS accepts a PED format supported by MERLIN or PLINK software to represent phenotypes. For example, the example.ped file and example.dat file can represent the phenotypes and corresponding column name (from 6th column and after).

$ head example.ped
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

$ cat example.dat
A DISEASE
T QT
T AGE

EPACTS also accept a PED format with header information. The above file can be combined into one file as follows

$ head data/1000G_dummy_pheno.ped
#FAM_ID IND_ID FAT_ID MOT_ID SEX DISEASE QT AGE
13281 NA12344 NA12347 NA12348 1 1 94.17 66.1
13281 NA12347 0 0 1 1 109.54 44.0
13281 NA12348 0 0 2 2 119.40 46.6
1328 NA06984 0 0 1 2 87.72 39.3
1328 NA06989 0 0 2 1 100.60 41.7
1328 NA12329 NA06984 NA06989 2 1 100.85 46.4
13291 NA06986 0 0 1 2 91.94 61.9
13291 NA06995 NA07435 NA07037 1 2 104.36 57.4
13291 NA06997 NA06986 NA07045 2 2 107.53 53.1

The column names can be used to identify the names of phenotypes and covariates in the analysis.

== Frequently Asked Questions ==
=== Installation ===
# How should I install EPACTS?
#* See [[EPACTS#Installation_Details | Installation Details]]
# I am having the following error message '''configure: error: libR.{so,a} was not found. Please install it at http://www.r-project.org/ first'''. What do I have to do?
#* First, you need to find out where R was installed. Try to type "locate libR.so" and see if it returns anything
#* If "locate libR.so" returns you something, as explained [[EPACTS#Installation_Details | Installation Details]], try to add "LDFLAGS=-L/path/to/R/library" and rerun '''configure''' and '''make'''
#* If you cannot find libR.so, you make have to recompile R with --enable-R-shlib option as described in http://cran.r-project.org/doc/manuals/R-admin.html#Installation

=== Input Files ===
# What is VCF?
#* VCF refers to Variant Call Format
#* See [[http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 1000 Genomes wiki page]] for the detailed description of VCF format
# Should input VCF be compressed into certain format?
#* Correct. EPACTS assumes that VCF file is bgzipped and tabixed already.
#* See [[#VCF file for Genotypes]] for details.
# What are the additional requirements for input VCF file?
#* Input VCF file used for association mapping must contain individual genotype information at 10-th or higher order columns.
#* GT field must be encoded as haploid or diploid
#* Bi-allelic SNPs only : Currently EPACTS may not handle multi-allelic SNPs correctly.
#* If non-GT field is used, the field is considered as dosage and should be a single numeric value.
# What are the acceptable input format to encode phenotypes and covariates?
#* See [[#PED file for Phenotypes and Covariates]] for the detailed information
# How should I encode binary phenotypes?
#* If you encode your phenotypes into two different numeric values (e.g. 0/1 or 1/2), EPACTS will automatically recognize them as binary phenotypes and encode them into 1/2 values. Higher value will be considered as cases for case-control association
# How should I encode missing genotypes?
#* The default code missing phenotypes in EPACTS are 'NA'
#* One may use --missing option to specify different types of missing values
#* The encoding of missing genotypes follows the VCF specificiation
# How do I match the relationship between VCF and PED input files?
#* EPACTS will assume that the individual IDs in each VCF and PED file are unique, and they follow the saming convention. Thus, the individual IDs overlapping between VCF and PED files will be considered in the associations
# How the individuals with missing phenotypes are handled?
#* Currently, EPACTS will automatically remove the individuals without phenotypes or covariates. If one wants to use imputed covariates to increase sample size, the PED file must contain the imputed covariate values.
#* Markers with missing genotypes won't be discarded automatically. It can be explicitly discarded by --minCallRate option when performing association

=== Output Files ===
# Which output files should I be looking at?
#* [[#Output Text of Top Associations]] is the key file to look at the individual top associations
#* [[#Q-Q plot of test statistics (stratified by MAF)]] will be important to see the global distribution of test statistics and examine if there are apparent inflation of test statistics
#* [[#Manhattan Plot of Test Statistics]] will inform us the genome-wide distribution of association signals
#* [[#Output Text of All Test Statistics]] will contain the full information of test results across all units tested
# The Q-Q and Manhattan plots cannot be found. Why?
#* It is probably because gnuplot 4.2 or higher is not installed in your system, or they are included but cannot be found in your ${PATH}. Please visit [[http://gnuplot.info/ GNUPLOT web page]] for installation.
# How can I read the EMMAX kinship file from produced from EPACTS?
# * You can run the following command to dump your kinship matrix into a human-readable text format.
$(EPACTS_DIR)/bin/epacts/bin/pEmmax kin-util --kinf [input.kinf] --outf [output.prefix] --dump

=== More questions ===
# If you have more questions, please contact [[mailto:hmkang@umich.edu Hyun Min Kang]].

== Detailed Options ==

The detailed options can viewed by running the following commands
${EPACTS_DIR}/bin/epacts -man (for overall structure)
${EPACTS_DIR}/bin/epacts single -man (for single variant test)
${EPACTS_DIR}/bin/epacts group -man (for groupwise test)
${EPACTS_DIR}/bin/epacts anno -man (for annotation)
${EPACTS_DIR}/bin/epacts plot -man (for QQ and Manhattan plot)
${EPACTS_DIR}/bin/epacts zoom -man (for zoom plot)
${EPACTS_DIR}/bin/epacts meta -man (for meta-analysis)
${EPACTS_DIR}/bin/epacts make-group -man (for creating gene group)

== Implementing Additional Statistical Tests ==

In order to add additional statistical test to EPACTS, the following procedure are recommended

# Create a file named 'single.[testname].R' for single variant test or 'gene.[testname].R' for gene-level test under ${EPACTS_DIR}/share/EPACTS/
# Test your implementation using --test [testname] option to perform sanity check and debugging
# If you want to add your test in the official in-house version, please send your code to Hyun

Below is an example of a single variant test implementation ( single.q.lm.R )
## Core functions of EPACTS to perform association

##################################################################
## SINGLE VARIANT TEST
## INPUT VARIABLES:
## n : total # of individuals
## NS : number of called samples
## AC : allele count
## MAF : minor allele frequency
## vids : indices from 1:nrow(NS) after AF/AC threshold
## genos : genotype matrix (after AF/AC threshold)
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional columns to add
## cname : column names for additional columns
##################################################################

## single.lm() : Use built-in lm() function to perform association
## KEY FEATURES : SIMPLE, BUT MAY BE SLOW
## GOOD SNIPPLET TO START A NEW FUNCTION
## TRAITS : QUANTITATIVE
## RETURNS : PVALUE, BETA, SEBETA, TSTAT
## MISSING VALUES : IGNORED
single.q.lm <- function() {
cname <- c("BETA","SEBETA","TSTAT") # column names for additional variables in the EPACTS output
m <- nrow(genos)
p <- rep(NA,m)
add <- matrix(NA,m,3) ## BETA, SEBETA, TSTAT
if ( m > 0 ) {
for(i in 1:m) {
r <- summary(lm(pheno~genos[i,]+cov-1))$coefficients[1,] # run simple linear regression
p[i] <- r[4] # store p-value to p[i]
add[i,] <- r[1:3] # store additional variables to add[i,]
}
}
return(list(p=p,add=add,cname=cname))
}

As described in the comment, you may assume that the following variables are available for use for testing association across m markers
* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (m * 1 vector) : p-value matrix as output
* add (m * c matrix) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# AC : Non-ref allele count
# CALLRATE : Call rate = NS/n
# MAF : Minor allele frequency
# PVALUE : P-values
# Additional columns specified by return values 'add'

Below is an example of a gene-lvel variant test implementation ( single.q.lm.R )

##################################################################
## GENE-LEVEL BURDEN TEST
## INPUT VARIABLES:
## n : total # of individuals
## genos : genotype matrix for each gene
## NS : number of called samples for each marker
## AC : allele count for each marker
## MAC : minor allele count for each marker
## MAF : minor allele frequency
## vids : indices from 1:n after AF/AC threshold
## EXPECTED OUTPUT : list(p, addcols, addnames) for each genos row
## p : p-value
## add : additional column values
## cname : additional column names
##################################################################

## gene.q.reverse() : Reverse logistic regression
## KEY FEATURES : 0/1 collapsing variable ~ rare variants
## TRAITS : QUANTITATIVE (GAUSSIAN)
## RETURNS : PVALUE, BETA, SEBETA, ZSTAT
## MISSING VALUE : IMPUTED AS MAJOR ALLELES
gene.q.reverse <- function() {
cname <- c("BETA","SEBETA","ZSTAT")
m <- nrow(genos)
if ( m > 0 ) {
g <- as.double(colSums(genos,na.rm=T) > 0)
sg <- sum(g)
if ( ( sg > 0 ) && ( sg < n ) ) {
r <- glm(g~pheno+cov-1,family=binomial)
if ( ( r$converged ) && ( ! r$boundary ) ) {
return(list(p=summary(r)$coefficients[1,4],
add=summary(r)$coefficients[1,1:3],
cname=cname))
}
}
}
return(list(p=NA,add=rep(NA,3),cname=cname))
}

Similar to gene-level test, you may assume the following variables exist for testing A SINGLE GENE. Note that M is the number of markers spanning the gene region

* n (scalar) : total number of individuals
* NS (M * 1 vector) : Number of called samples for each marker
* AC (M * 1 vector) : Non-reference allele count for each marker
* MAC (M * 1 vector) : Minor allele count
* MAF (M * 1 vector) : Minor allele frequency
* vids (m * 1 vector) : indices of markers passing the inclusion criteria (e.g. MAF threshold) among 1:M
* genos (m * n matrix) : genotype matrix as a input for association test

The output variables to generate is as follows
* p (scalar) : p-value matrix as output
* add (c * 1 vector) : additional columns as output of test (such as SCORE, BETA, etc)
* cname (c * 1 vector) : column names of add

In the output files, the following columns will be displayed
# MARKER : Marker ID
# NS : Number of called samples
# MAF_BURDEN : MAF of 0/1 collapsing variables (existence of rare variants)
# NUM_ALL_VARS : Number of all variants within the gene
# NUM_RARE_VARS : Number of rare variants below the max-MAF threshold
# NUM_SING_VARS : Number of singleton variants
# PVALUE : P-value from the test
# Additional columns specified by return values 'add'

== Full ChangeLog ==
* July 10th, 2014 : EPACTS v3.2.6 release
** Minor bug fix in epacts-make-kin
* March 11th, 2014 : EPACTS v3.2.5 release
** EMMAX-SKAT is implemented with major bug fix
* November 21th, 2013 : EPACTS v3.2.4 release
** Fixed a number of minor bugs
** Some known bugs still exist
*** SKAT-O Lambda eigenvalue error. This happenes in a particular context but haven't nailed down a way to prevent it yet.
*** EMMAX has case and control frequency flipped.
* EMMAX test has a silly known bug with case / ctrl frequency is flipped
* March 25th, 2013 : EPACTS v3.2.3 release
** Relaxed the checking of low-rank matrix in SKAT tests (to avoid unncessary skipping of genes)
* March 13th, 2013 : EPACTS v3.2.2 release
** Fixed an error which occasionally report mismatches in the number of samples
* March 9th, 2013 : EPACTS v3.2.1 release
**Fixed errors in loading the dynamic library
** Fixed errors in SKAT-O (thanks to Anubha Mahajan and Jason Flannick)
** Fixed bugs in emmax-CMC
** Added emmax-SKAT (contributed by Seunngeun Lee)
** And additional minor bug fixes
* February 28th, 2013 : EPACTS v3.2.0 release
** R package installation bug (for some users) was fixed
** A bug in the MAF error for high frequency variants (AF>0.25) was now fixed
** SKAT version is updated to 0.81
** --bprange option is added to allow testing for small region size
** Additional minor bug fixes
* December 4th, 2012 : EPACTS v3.1.0 release
** Removed dependency on libR.so
** Additional minor bug fixes
** --bprange option is added to allow testing for small region size
** November 25th, 2012 : EPACTS v3.0.0 release
** Restructured with source code release (with autoconf / automake / libtools)
** Added zoom plot feature
** FRAC_BURDEN keyword was replace to FRAC_WITH_RARE for groupwise testing
* October 26th, 2012 : EPACTS v2.2.0-beta is released with the following updates
** Added --max-mac option
** Fixed Firth's bias-corrected test (by Clement Ma)
** Added more informative warning messages when index files do not exist
** Fixed the bug in the epacts-plot in plotting ties
** Fixed errors in the MAF estimates per case and control
** Fixed bug in --minRSQ option
* September 28, 2012 : EPACTS v2.11-beta is released with the following updates
** Counts and allele frequencies for case/control added for binary tests
** --max-maf parameter is added
** Fixed EMMAX error in MAF in the output
** More informative error messages
* September 27, 2012 : EPACTS v2.1-beta is released with the following updates
** EMMAX interface is changed. --kinOnly option is related with a new command '''make-kin'''
** SKAT-O is upgraded to version 0.77 with additional configurable parameter settings
** Some parameter names are renamed (e.g. --min-maf, --min-mac)
** Many minor bugs are fixed
* Jul 6, 2012 : EPACTS v2.01-beta is released with the following updates
** SKAT-O is upgraded to version 0.76
** Fixed minor bugs in option names (Thanks to Xueling Sim)
* Jul 3, 2012 : EPACTS v2.0-beta is released with the following updates
** Major restructuring of the software
** Annotation software is switched with built-in application
** Addition of SKAT-O and EMMAX burden test
** Minor bug fixes
* Apr 8, 2012 : EPACTS v1.2-alpha is released with the following updates, in addition to the following updates
** EMMAX bug in handling covariates was fixed
** Variable Threshold Test is added
** Variable Threshold Test with genomic score (e.g. GERP or PhyloP) is added.
* Apr 4, 2012 : EPACTS v1.1-alpha is released with the following updates, in addition to minor updates
** EMMAX burden test (Hyun Min Kang)
** Likelihood ratio test (Clement Ma)
** Updated version of Firth bias-corrected likelihood ratio test (Clement Ma)
** Updated version of EMMAX single variant test (Hyun Min Kang)
* Mar 29, 2012 : EPACTS v1.0-alpha is released

VerifyBamID

2015-10-31T22:32:26Z

Hmkang: /* Acknowledgements */

[[Category:Software|VerifyBamID]]

'''verifyBamID''' is a software that verifies whether the reads in particular file match previously known genotypes for an individual (or group of individuals), and checks whether the reads are contaminated as a mixture of two samples. '''verifyBamID''' can detect sample contamination and swaps when external genotypes are available. When external genotypes are not available, '''verifyBamID''' still robustly detects sample swaps.

== Download verifyBamID ==

To get a copy of verifyBamId, go to: https://github.com/statgen/verifyBamID/releases

Select the latest release and download in one of 3 ways:
# Binary expected to run in Ubuntu x64 platform. In other platforms, please download the source distribution and build it.
#* verifyBamID.#.#.#.gz
#* You will need to run "gunzip" on the .gz file
# Souce Code including libStatGen (uses a fixed version of libStatGen)
#* verifyBamIDLibStatGen.#.#.#.tgz
#* Run "tar xvf" on this file. Cd into the resulting directory & type make.
# Source Code without libStatGen (allows alternative/newer versions of libStatGen)
#* Source code (tar.gz) or Source code (zip)
#* You will need to download libStatGen separately if you do not already have it.

To get a copy of older releases go to the [http://www.sph.umich.edu/csg/kang/verifyBamID/download VerifyBamID Download] download page.

== Join in verifyBamID mailing list ==

Please join in the [http://groups.google.com/group/verifybamid VerifyBamID Google Group] to ask / discuss / comment about verifyBamID.

== What's new ==

(2014/02/13)
* Put verifyBamID in github.
* Added PhoneHome/Version Checking to VerifyBamID

(2012/06/20)
* Fixed a bug of incorrect estimate of contamination when --chip-full option was used (Thanks to Richard Smith)
* Fixed a bug of incorrect per-readgroup output in --chip-* parameter

(2012/05/24)
* Fixed a bug of incorrect per-readgroup output (Thanks to Matthew Flickinger)
* '''(IMPORTANT)''' Add an option to remove either side of overlapping fragment. This option is turned on by default, and can be turned off usig --ignoreOverlapPair. If your sequence data has very short insert size, this update may increase the sensitivity of estimated contamination.
* Changes in the directory structure and Makefile

(2012/05/18) The new release of verifyBamID have undergone major change since the last version (as of 2011 April). Here are the highlights
* The genotype / allele frequency file is now based on VCF format rather than PLINK format.
* The reference sequence information is no longer required
* Uses Brent's method for precise estimation of contamination parameters
* Generate the depth distribution statistics.
* Estimated reference-bias parameters (useful mostly for ABI SOLiD sequence data)

== Build verifyBamID ==

The binary download of verifyBamID is available. You may use that version in Ubuntu 64-bit platform.

If you download the source that includes libStatGen:
tar xvf verifyBamIDLibStatGen.#.#.#.tgz
cd verifyBamID_#.#.#
make
Executable: verifyBamID/bin/verifyBamID

If you download the source without libStatGen:
tar xvf verifyBamID-#.#.#.tar.gz
cd verifyBamID-1.1.0
make cloneLib (if ../libStatGen does not exist)
make
Executable: ./bin/verifyBamID

Note that '''make cloneLib''' command will create a directory ../libStatGen under your verifyBamID directory, and '''make''' will create binary of verifyBamID under verifyBamID/bin/

If you have a different version of libStatGen at that path, then skip the cloneLib step. If the libStatGen you want to use is at a different location then update verifyBamID's Makefile.inc. Replace: LIB_PATH_VERIFY_BAM_ID ?= $(LIB_PATH_GENERAL) with
LIB_PATH_VERIFY_BAM_ID = /path/to/libStatGen

verifyBamID is designed to be reasonably portable.

However, since development occurs only on Ubuntu (9.10-13.10) x86 and x64 platforms, and later, there are likely other portability issues.

== Basic Usage ==

A key step in any genetic analysis is to verify whether data being generated matches expectations. ''verifyBamID'' checks whether reads in a BAM file match previous genotypes for a specific sample. In addition, it detects possible sample mixture from population allele frequency only, which can be particularly useful when the genotype data is not available.

Using a mathematical model that relates observed sequence reads to an hypothetical true genotype, ''verifyBamID'' tries to decide whether sequence reads match a particular individual or are more likely to be contaminated (including a small proportion of foreign DNA), derived from a closely related individual, or derived from a completely different individual.

== Basic Usage Example ==

Here is a typical command line:

verifyBamID --vcf [input.vcf] --bam [input.bam] --out [output.prefix] --verbose --ignoreRG

where
[input.bam] is a BAM (Binary Alignment Map) file of a sequence reads
[input.vcf] is input VCF file containing individual genotypes or AF or AC/AN fields in the INFO field. gzipped VCF is also allowed.
[outPrefix] is output prefix of output files - [outPrefix].{selfRG,selfSM,bestRG,bestSM,depthRG,depthSM} will be created.

More detailed description of command line input is below

== Preparing input files ==

verifyBamID requires two input files - VCF file containing external genotypes or allele frequency information, and the BAM file.

=== VCF input genotype file ===

The input VCF file contains (1) external genotype information and/or (2) allele frequency information as AF entry or AC/AN entries in the INFO field. (See [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 | VCF specification] for further details). If neither information is provided, verifyBamID will not work properly.

If external genotype information is provided, sequence+array method will identify contamination and sample swaps by comparing the concordance between the external genotypes and the sequence reads. Additionally, sequence-only method will provide additional contamination estimates by modeling the sequence reads as mixture of two unknown samples based on the allele frequency information in the VCF file.

Input VCF file needs to meet several additional contraints need to meet in order to properly run verifyBamID.
* The VCF is assumed to be well-formed. For example, verifyBamID does not check whether REF allele actually matches with reference sequence.
* The VCF should only contain SNPs. Current version of verifyBamID does not accept INDELs, MNPs, Structural Variations, or other complex variants.
* The individual IDs in the VCF file, must be identical with the individual identifier in the BAM file. Otherwise, --smID option can override the sample ID information of the BAM file to the ID that matches to the individual IDs in the VCF file.
* IMPORTANT : For targeted sequencing data, it is important to subselect the markers to only include on-target markers in the genotype file. Off-target markers are not likely to have multiple non-duplicated reads at the marker position, and it may create artifacts in the analysis due to overlapping fragments.
* Currently, verifyBamID takes only autosomal chromosomes as input VCF.

An example input VCF file (without external genotype) is provided below. Note that AC and AC entries exists in the INFO field for the allele frequency information.

#CHROM POS ID REF ALT QUAL FILTER INFO
20 61651 SNP20-9651 C A . PASS CR=99.86851;GentrainScore=0.7055;HW=0.077647716;AN=2180;AC=11
20 63231 SNP20-11231 T G . PASS CR=99.93036;GentrainScore=0.7837;HW=0.035481825;AN=2182;AC=275
20 63244 rs6139074 A C . PASS CR=98.893394;GentrainScore=0.8001;HW=7.327299E-7;AN=2162;AC=501
20 63799 rs1418258 C T . PASS CR=99.75217;GentrainScore=0.8170;HW=0.6653377;AN=2182;AC=881

=== Input BAM file ===

verifyBamID requires a sorted, indexed, base quality recalibrated, and duplication-marked BAM file. It also requires to contain "@RG" header lines to annotation different readGroups (sequencing runs and lanes). The SM tag in the "@RG" header should match with one of the genotyped sample. Otherwise, verifyBamID may not be able to test whether the sequenced sample matches with genotyped sample, but will try to detect sample mixture from allele frequency, and will try to detect the best-matching sample among the genotyped sample.

== What the default option does ==

The default option of '''verifyBamID''' is the recommended setting for the most sequencing studies to provide a rapid and informative response. The default option provides the following features:
* --free-mix is turned on for estimating contamination using sequence-only method
* --chip-mix is turned on for estimating contamination or swap using sequence+array method, if the external genotype file is provided in the VCF
* --self is turnd on : The default option does not try to compare the sequence reads to identify the best matching individual (which is possible with --best option). It only compares with the external genotypes from the same individual to the sequenced individual.
* --maxDepth 20 is used without --precise option : The default option is intended for whole genome low coverage sequencing. For the targeted exome sequencing, --maxDepth 1000 and --precise is recommended.
* --ignoreRG is not a default option, but a recommended option, when you want to check the contamination for the entire BAM rather than examining each read group separately. This option will increase the computational efficiency especially in the case whether the sequence reads are multiplexed across many sequencing runs.

== Interpreting output files ==

See also [[Understanding VerifyBamID output]].

=== Output files ===
When verifyBamID runs successfully, the following sets of files may be generated.
* [outPrefix].selfSM - Per-sample statistics describing how well the sample matches to the annotated sample.
* [outPrefix].depthSM - The depth distribution of the sequence reads per sample
* [outPrefix].selfRG - Per-readGroup statistics describing how well each lane matches to the annotated sample. (available only without --ignoreRG option)
* [outPrefix].depthRG - The depth distribution of the sequence reads per readGroup. (available only without --ignoreRG option)
* [outPrefix].bestSM - Per-sample best-match statistics with best-matching sample among the genotyped sample (available only with --best option)
* [outPrefix].bestRG - Per-readgroup best-match statistics with best-matching sample among the genotyped sample (available only with --best and without --ignoreRG option)

=== Column information in the output files ===
The .selfSM/.selfRG/.bestSM/.bestRG files have the following 19 columns per sample, or per readgroup (lane).

# SEQ_SM : Sample ID of the sequenced sample. Obtained from @RG header / SM tag in the BAM file
# RG : ReadGroup ID of sequenced lane. For [outPrefix].selfSM and [outPrefix].bestSM, these values are "ALL"
# CHIP_ID : Sample ID compared to in the genotype file. For [outPrefix].selfRG and [outPrefix].selfSM, these values should be identical to [SEQ_SM] or "NA" if the genotype of sequenced samples are unavailable. For [outPrefix].bestRG and [outPrefix].bestSM, these values should be the ID of best-matching sample among the genotype files compared to.
# # SNPs : # of SNPs passing the criteria from the VCF file
# # READS : Total # of reads loaded from the BAM file
# # AVG_DP : Average sequencing depth at the sites in the VCF file
# FREEMIX : Sequence-only estimate of contamination (0-1 scale)
# FREELK1 : Maximum log-likelihood of the sequence reads given estimated contamination under sequence-only method
# FREELK0 : Log-likelihood of the sequence reads given no contamination under sequence-only method
# FREE_RH : Estimated reference bias parameter Pr(refBase|HET) (when --free-refBias or --free-full is used)
# FREE_RA : Estimated reference bias parameter Pr(refBase|HOMALT) (when --free-refBias or --free-full is used)
# CHIPMIX : Sequence+array estimate of contamination (NA if the external genotype is unavailable) (0-1 scale)
# CHIPLK1 : Maximum log-likelihood of the sequence reads given estimated contamination under sequence+array method (NA if the external genotypes are unavailable)
# CHIPLK0 : Log-likelihood of the sequence reads given no contamination under sequence+array method (NA if the external genotypes are unavailable)
# CHIP_RH : Estimated reference bias parameter Pr(refBase|HET) (when --chip-refBias or --chip-full is used)
# CHIP_RA : Estimated reference bias parameter Pr(refBase|HOMALT) (when --chip-refBias or --chip-full is used)
# DPREF : Depth (Coverage) of HomRef site (based on the genotypes of (SELF_SM/BEST_SM), passing mapQ, baseQual, maxDepth thresholds.
# RDPHET : DPHET/DPREF, Relative depth at Heterozygous site.
# RDPALT : DPHET/DPREF, Relative depth at HomAlt site.

=== A guideline to interpret output files ===

verifyBamID provides a series of information that is informative to determine whether the sample is possibly contaminated or swapped, but there is no single criteria that works for every circumstances. There are a few unmodeled factor in the estimation of [SELF-IBD]/[BEST-IBD] and [%MIX], so please note that the MLE estimation may not always exactly match to the true amount of contamination. Here we provide a guideline to flag potentially contaminated/swapped samples

* Each sample or lane can be checked in this way. When [CHIPMIX] >> 0.02 and/or [FREEMIX] >> 0.02, meaning 2% or more of non-reference bases are observed in reference sites, we recommend to examine the data more carefully for the possibility of contamination.
* We recommend to check each lane for the possibility of sample swaps. When [CHIPMIX] ~ 1 AND [FREEMIX] ~ 0, then it is possible that the sample is swapped with another sample. When [CHIPMIX] ~ 0 in .bestSM file, [CHIP_ID] might be actually the swapped sample. Otherwise, the swapped sample may not exist in the genotype data you have compared.
* When genotype data is not available but allele-frequency-based estimates of [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, then it is possible that the sample is contaminated with other sample. We recommend to use per-sample data rather than per-lane data for checking this for low coverage data, because the inference will be more confident when there are large number of bases with depth 2 or higher.

== Command Line Options ==

The following parameters are available. Ones with "[]" are in effect:

Available Options
Input Files : --vcf [], --bam [], --subset [],
--smID []
VCF analysis options : --genoError [1.0e-03],
--minAF [0.01],
--minCallRate [0.50]
Individuals to compare with chip data : --site, --self, --best
Chip-free optimization options : --free-none, --free-mix [ON],
--free-refBias, --free-full
With-chip optimization options : --chip-none, --chip-mix [ON],
--chip-refBias, --chip-full
BAM analysis options : --ignoreRG, --ignoreOverlapPair,
--noEOF, --precise, --minMapQ [10],
--maxDepth [20], --minQ [13],
--maxQ [40], --grid [0.05]
Modeling Reference Bias : --refRef [1.00], --refHet [0.50],
--refAlt [0.00]
Output options : --out [], --verbose
PhoneHome : --noPhoneHome,
--phoneHomeThinning [50]

Each option provides the following features:
* --vcf : specify required VCF file
* --bam : specify required BAM file (indexed with .bam.bai or .bai file)
* --subset : list of individual IDs to calculate the allele frequency. All individuals are used if unspecified
* --smID : If the individual ID in the BAM file and VCF file does not match, substitute the BAM file's ID into the specified argument
* --genoError : error rate of the external genotype file
* --minAF : minimum allele frequency of the markers to include
* --minAF : minimum call rate of the markers to include
* --site : If set, use only site information in the VCF and do not compare with the actual genotypes
* --self : Only compare the ID-matching individuals between the VCF and BAM file
* --best : Find the best matching individuals (.bestSM and .bestRG files will be produced). This option is substantially longer than the default option
* --free-none : Do not perform sequence-only method to estimate parameters
* --free-mix : (default) Estimate contamination using sequence-only method with Brent's single dimensional optimization.
* --free-refBias : Estimate the reference bias parameters using sequence-only method with Simplex method
* --free-full : Estimate both reference bias parameters and the contamination parameters using sequence-only method
* --chip-none : Do not perform sequence+array method to estimate parameters
* --free-mix : (default) Estimate contamination using sequence+array method with Brent's single dimensional optimization.
* --free-refBias : Estimate the refernece bias parameters using sequence+array method with Simplex method
* --free-full : Estimate both reference bias parameters and the contamination parameters using sequence+array method
* --ignoreRG : ignore the read grouup level comparison and compare samples only (recommended for an expedited run)
* --ignoreOverlapPair : ignore overlapping pair end fragment covering the same base. Disabling this option may decrease the sensitivity of the method when the insert size is short (with slight gain in the computational speed)
* --noEOF : do not check the EOF marker of the BAM file (for earlier version of BAM)
* --precise : calculate the likelihood in log-scale for high-depth data (recommended when --maxDepth is greater than 20. Can be a little bit slower)
* --minMapQ : minimum mapping quality of the sequence reads to compare
* --minQ : minimum base quality to include
* --maxQ : maximum base quality to cap
* --grid : the grid interval to search the optimum before running Brent's algorithm.
* --refRef : Initial Pr(refBase|HOMREFGeno) parameter
* --refHet : Initial Pr(refBase|HETGeno) parameter
* --refAlt : Initial Pr(refBase|HOMALTGeno) parameter
* --out : output file prefix (required)
* --verbose : print the progress of the method on the screeen
{{PhoneHomeParameters|hdr=====|bullet=1}}

== Principle of Operation ==

Each read group in a BAM file is evaluated independently. This means that in file with multiple read groups, problems will be flagged at the read group level (a plus). However, it also means that it might be hard to discern the correct assignment of read groups with very little data.

For each aligned base that overlaps a known genotype, we calculate the probability the probability that it was derived from a particular known genotype. This comparison considers only bases that overlap previously known genotypes and that meet the base quality and mapping quality thresholds.

Each individual in a pedigree has a different combination of genotypes, and bamGenotypeCheck will systematically search for the individual whose genotypes best match the observed read data.

For more about the technical details, see the page [[Verifying Sample Identities - Implementation]]

== Reference ==

Please cite the following paper:

G. Jun, M. Flickinger, K. N. Hetrick, Kurt, J. M. Romm, K. F. Doheny, G. Abecasis, M. Boehnke,and H. M. Kang, ''Detecting and Estimating Contamination of Human DNA Samples in Sequencing and Array-Based Genotype Data'', American journal of human genetics doi:10.1016/j.ajhg.2012.09.004 (volume 91 issue 5 pp.839 - 848)

== Contamination in Array Data ==

[[VerifyIDintensity]] or [[BAFRegress]] can estimate sample contamination from Illumina genotype array data.

== Acknowledgements ==

VerifyBamID is a result from collaborative effort by Hyun Min Kang, Goo Jun, Matthew Flickinger, Mary Kate Wing, Goncalo Abecasis, and Michael Boehnke. Please email to Hyun Min Kang [hmkang@umich.edu ] for any questions.

VerifyBamID

2015-10-31T22:30:51Z

Hmkang: /* Acknowledgements */

[[Category:Software|VerifyBamID]]

'''verifyBamID''' is a software that verifies whether the reads in particular file match previously known genotypes for an individual (or group of individuals), and checks whether the reads are contaminated as a mixture of two samples. '''verifyBamID''' can detect sample contamination and swaps when external genotypes are available. When external genotypes are not available, '''verifyBamID''' still robustly detects sample swaps.

== Download verifyBamID ==

To get a copy of verifyBamId, go to: https://github.com/statgen/verifyBamID/releases

Select the latest release and download in one of 3 ways:
# Binary expected to run in Ubuntu x64 platform. In other platforms, please download the source distribution and build it.
#* verifyBamID.#.#.#.gz
#* You will need to run "gunzip" on the .gz file
# Souce Code including libStatGen (uses a fixed version of libStatGen)
#* verifyBamIDLibStatGen.#.#.#.tgz
#* Run "tar xvf" on this file. Cd into the resulting directory & type make.
# Source Code without libStatGen (allows alternative/newer versions of libStatGen)
#* Source code (tar.gz) or Source code (zip)
#* You will need to download libStatGen separately if you do not already have it.

To get a copy of older releases go to the [http://www.sph.umich.edu/csg/kang/verifyBamID/download VerifyBamID Download] download page.

== Join in verifyBamID mailing list ==

Please join in the [http://groups.google.com/group/verifybamid VerifyBamID Google Group] to ask / discuss / comment about verifyBamID.

== What's new ==

(2014/02/13)
* Put verifyBamID in github.
* Added PhoneHome/Version Checking to VerifyBamID

(2012/06/20)
* Fixed a bug of incorrect estimate of contamination when --chip-full option was used (Thanks to Richard Smith)
* Fixed a bug of incorrect per-readgroup output in --chip-* parameter

(2012/05/24)
* Fixed a bug of incorrect per-readgroup output (Thanks to Matthew Flickinger)
* '''(IMPORTANT)''' Add an option to remove either side of overlapping fragment. This option is turned on by default, and can be turned off usig --ignoreOverlapPair. If your sequence data has very short insert size, this update may increase the sensitivity of estimated contamination.
* Changes in the directory structure and Makefile

(2012/05/18) The new release of verifyBamID have undergone major change since the last version (as of 2011 April). Here are the highlights
* The genotype / allele frequency file is now based on VCF format rather than PLINK format.
* The reference sequence information is no longer required
* Uses Brent's method for precise estimation of contamination parameters
* Generate the depth distribution statistics.
* Estimated reference-bias parameters (useful mostly for ABI SOLiD sequence data)

== Build verifyBamID ==

The binary download of verifyBamID is available. You may use that version in Ubuntu 64-bit platform.

If you download the source that includes libStatGen:
tar xvf verifyBamIDLibStatGen.#.#.#.tgz
cd verifyBamID_#.#.#
make
Executable: verifyBamID/bin/verifyBamID

If you download the source without libStatGen:
tar xvf verifyBamID-#.#.#.tar.gz
cd verifyBamID-1.1.0
make cloneLib (if ../libStatGen does not exist)
make
Executable: ./bin/verifyBamID

Note that '''make cloneLib''' command will create a directory ../libStatGen under your verifyBamID directory, and '''make''' will create binary of verifyBamID under verifyBamID/bin/

If you have a different version of libStatGen at that path, then skip the cloneLib step. If the libStatGen you want to use is at a different location then update verifyBamID's Makefile.inc. Replace: LIB_PATH_VERIFY_BAM_ID ?= $(LIB_PATH_GENERAL) with
LIB_PATH_VERIFY_BAM_ID = /path/to/libStatGen

verifyBamID is designed to be reasonably portable.

However, since development occurs only on Ubuntu (9.10-13.10) x86 and x64 platforms, and later, there are likely other portability issues.

== Basic Usage ==

A key step in any genetic analysis is to verify whether data being generated matches expectations. ''verifyBamID'' checks whether reads in a BAM file match previous genotypes for a specific sample. In addition, it detects possible sample mixture from population allele frequency only, which can be particularly useful when the genotype data is not available.

Using a mathematical model that relates observed sequence reads to an hypothetical true genotype, ''verifyBamID'' tries to decide whether sequence reads match a particular individual or are more likely to be contaminated (including a small proportion of foreign DNA), derived from a closely related individual, or derived from a completely different individual.

== Basic Usage Example ==

Here is a typical command line:

verifyBamID --vcf [input.vcf] --bam [input.bam] --out [output.prefix] --verbose --ignoreRG

where
[input.bam] is a BAM (Binary Alignment Map) file of a sequence reads
[input.vcf] is input VCF file containing individual genotypes or AF or AC/AN fields in the INFO field. gzipped VCF is also allowed.
[outPrefix] is output prefix of output files - [outPrefix].{selfRG,selfSM,bestRG,bestSM,depthRG,depthSM} will be created.

More detailed description of command line input is below

== Preparing input files ==

verifyBamID requires two input files - VCF file containing external genotypes or allele frequency information, and the BAM file.

=== VCF input genotype file ===

The input VCF file contains (1) external genotype information and/or (2) allele frequency information as AF entry or AC/AN entries in the INFO field. (See [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 | VCF specification] for further details). If neither information is provided, verifyBamID will not work properly.

If external genotype information is provided, sequence+array method will identify contamination and sample swaps by comparing the concordance between the external genotypes and the sequence reads. Additionally, sequence-only method will provide additional contamination estimates by modeling the sequence reads as mixture of two unknown samples based on the allele frequency information in the VCF file.

Input VCF file needs to meet several additional contraints need to meet in order to properly run verifyBamID.
* The VCF is assumed to be well-formed. For example, verifyBamID does not check whether REF allele actually matches with reference sequence.
* The VCF should only contain SNPs. Current version of verifyBamID does not accept INDELs, MNPs, Structural Variations, or other complex variants.
* The individual IDs in the VCF file, must be identical with the individual identifier in the BAM file. Otherwise, --smID option can override the sample ID information of the BAM file to the ID that matches to the individual IDs in the VCF file.
* IMPORTANT : For targeted sequencing data, it is important to subselect the markers to only include on-target markers in the genotype file. Off-target markers are not likely to have multiple non-duplicated reads at the marker position, and it may create artifacts in the analysis due to overlapping fragments.
* Currently, verifyBamID takes only autosomal chromosomes as input VCF.

An example input VCF file (without external genotype) is provided below. Note that AC and AC entries exists in the INFO field for the allele frequency information.

#CHROM POS ID REF ALT QUAL FILTER INFO
20 61651 SNP20-9651 C A . PASS CR=99.86851;GentrainScore=0.7055;HW=0.077647716;AN=2180;AC=11
20 63231 SNP20-11231 T G . PASS CR=99.93036;GentrainScore=0.7837;HW=0.035481825;AN=2182;AC=275
20 63244 rs6139074 A C . PASS CR=98.893394;GentrainScore=0.8001;HW=7.327299E-7;AN=2162;AC=501
20 63799 rs1418258 C T . PASS CR=99.75217;GentrainScore=0.8170;HW=0.6653377;AN=2182;AC=881

=== Input BAM file ===

verifyBamID requires a sorted, indexed, base quality recalibrated, and duplication-marked BAM file. It also requires to contain "@RG" header lines to annotation different readGroups (sequencing runs and lanes). The SM tag in the "@RG" header should match with one of the genotyped sample. Otherwise, verifyBamID may not be able to test whether the sequenced sample matches with genotyped sample, but will try to detect sample mixture from allele frequency, and will try to detect the best-matching sample among the genotyped sample.

== What the default option does ==

The default option of '''verifyBamID''' is the recommended setting for the most sequencing studies to provide a rapid and informative response. The default option provides the following features:
* --free-mix is turned on for estimating contamination using sequence-only method
* --chip-mix is turned on for estimating contamination or swap using sequence+array method, if the external genotype file is provided in the VCF
* --self is turnd on : The default option does not try to compare the sequence reads to identify the best matching individual (which is possible with --best option). It only compares with the external genotypes from the same individual to the sequenced individual.
* --maxDepth 20 is used without --precise option : The default option is intended for whole genome low coverage sequencing. For the targeted exome sequencing, --maxDepth 1000 and --precise is recommended.
* --ignoreRG is not a default option, but a recommended option, when you want to check the contamination for the entire BAM rather than examining each read group separately. This option will increase the computational efficiency especially in the case whether the sequence reads are multiplexed across many sequencing runs.

== Interpreting output files ==

See also [[Understanding VerifyBamID output]].

=== Output files ===
When verifyBamID runs successfully, the following sets of files may be generated.
* [outPrefix].selfSM - Per-sample statistics describing how well the sample matches to the annotated sample.
* [outPrefix].depthSM - The depth distribution of the sequence reads per sample
* [outPrefix].selfRG - Per-readGroup statistics describing how well each lane matches to the annotated sample. (available only without --ignoreRG option)
* [outPrefix].depthRG - The depth distribution of the sequence reads per readGroup. (available only without --ignoreRG option)
* [outPrefix].bestSM - Per-sample best-match statistics with best-matching sample among the genotyped sample (available only with --best option)
* [outPrefix].bestRG - Per-readgroup best-match statistics with best-matching sample among the genotyped sample (available only with --best and without --ignoreRG option)

=== Column information in the output files ===
The .selfSM/.selfRG/.bestSM/.bestRG files have the following 19 columns per sample, or per readgroup (lane).

# SEQ_SM : Sample ID of the sequenced sample. Obtained from @RG header / SM tag in the BAM file
# RG : ReadGroup ID of sequenced lane. For [outPrefix].selfSM and [outPrefix].bestSM, these values are "ALL"
# CHIP_ID : Sample ID compared to in the genotype file. For [outPrefix].selfRG and [outPrefix].selfSM, these values should be identical to [SEQ_SM] or "NA" if the genotype of sequenced samples are unavailable. For [outPrefix].bestRG and [outPrefix].bestSM, these values should be the ID of best-matching sample among the genotype files compared to.
# # SNPs : # of SNPs passing the criteria from the VCF file
# # READS : Total # of reads loaded from the BAM file
# # AVG_DP : Average sequencing depth at the sites in the VCF file
# FREEMIX : Sequence-only estimate of contamination (0-1 scale)
# FREELK1 : Maximum log-likelihood of the sequence reads given estimated contamination under sequence-only method
# FREELK0 : Log-likelihood of the sequence reads given no contamination under sequence-only method
# FREE_RH : Estimated reference bias parameter Pr(refBase|HET) (when --free-refBias or --free-full is used)
# FREE_RA : Estimated reference bias parameter Pr(refBase|HOMALT) (when --free-refBias or --free-full is used)
# CHIPMIX : Sequence+array estimate of contamination (NA if the external genotype is unavailable) (0-1 scale)
# CHIPLK1 : Maximum log-likelihood of the sequence reads given estimated contamination under sequence+array method (NA if the external genotypes are unavailable)
# CHIPLK0 : Log-likelihood of the sequence reads given no contamination under sequence+array method (NA if the external genotypes are unavailable)
# CHIP_RH : Estimated reference bias parameter Pr(refBase|HET) (when --chip-refBias or --chip-full is used)
# CHIP_RA : Estimated reference bias parameter Pr(refBase|HOMALT) (when --chip-refBias or --chip-full is used)
# DPREF : Depth (Coverage) of HomRef site (based on the genotypes of (SELF_SM/BEST_SM), passing mapQ, baseQual, maxDepth thresholds.
# RDPHET : DPHET/DPREF, Relative depth at Heterozygous site.
# RDPALT : DPHET/DPREF, Relative depth at HomAlt site.

=== A guideline to interpret output files ===

verifyBamID provides a series of information that is informative to determine whether the sample is possibly contaminated or swapped, but there is no single criteria that works for every circumstances. There are a few unmodeled factor in the estimation of [SELF-IBD]/[BEST-IBD] and [%MIX], so please note that the MLE estimation may not always exactly match to the true amount of contamination. Here we provide a guideline to flag potentially contaminated/swapped samples

* Each sample or lane can be checked in this way. When [CHIPMIX] >> 0.02 and/or [FREEMIX] >> 0.02, meaning 2% or more of non-reference bases are observed in reference sites, we recommend to examine the data more carefully for the possibility of contamination.
* We recommend to check each lane for the possibility of sample swaps. When [CHIPMIX] ~ 1 AND [FREEMIX] ~ 0, then it is possible that the sample is swapped with another sample. When [CHIPMIX] ~ 0 in .bestSM file, [CHIP_ID] might be actually the swapped sample. Otherwise, the swapped sample may not exist in the genotype data you have compared.
* When genotype data is not available but allele-frequency-based estimates of [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, then it is possible that the sample is contaminated with other sample. We recommend to use per-sample data rather than per-lane data for checking this for low coverage data, because the inference will be more confident when there are large number of bases with depth 2 or higher.

== Command Line Options ==

The following parameters are available. Ones with "[]" are in effect:

Available Options
Input Files : --vcf [], --bam [], --subset [],
--smID []
VCF analysis options : --genoError [1.0e-03],
--minAF [0.01],
--minCallRate [0.50]
Individuals to compare with chip data : --site, --self, --best
Chip-free optimization options : --free-none, --free-mix [ON],
--free-refBias, --free-full
With-chip optimization options : --chip-none, --chip-mix [ON],
--chip-refBias, --chip-full
BAM analysis options : --ignoreRG, --ignoreOverlapPair,
--noEOF, --precise, --minMapQ [10],
--maxDepth [20], --minQ [13],
--maxQ [40], --grid [0.05]
Modeling Reference Bias : --refRef [1.00], --refHet [0.50],
--refAlt [0.00]
Output options : --out [], --verbose
PhoneHome : --noPhoneHome,
--phoneHomeThinning [50]

Each option provides the following features:
* --vcf : specify required VCF file
* --bam : specify required BAM file (indexed with .bam.bai or .bai file)
* --subset : list of individual IDs to calculate the allele frequency. All individuals are used if unspecified
* --smID : If the individual ID in the BAM file and VCF file does not match, substitute the BAM file's ID into the specified argument
* --genoError : error rate of the external genotype file
* --minAF : minimum allele frequency of the markers to include
* --minAF : minimum call rate of the markers to include
* --site : If set, use only site information in the VCF and do not compare with the actual genotypes
* --self : Only compare the ID-matching individuals between the VCF and BAM file
* --best : Find the best matching individuals (.bestSM and .bestRG files will be produced). This option is substantially longer than the default option
* --free-none : Do not perform sequence-only method to estimate parameters
* --free-mix : (default) Estimate contamination using sequence-only method with Brent's single dimensional optimization.
* --free-refBias : Estimate the reference bias parameters using sequence-only method with Simplex method
* --free-full : Estimate both reference bias parameters and the contamination parameters using sequence-only method
* --chip-none : Do not perform sequence+array method to estimate parameters
* --free-mix : (default) Estimate contamination using sequence+array method with Brent's single dimensional optimization.
* --free-refBias : Estimate the refernece bias parameters using sequence+array method with Simplex method
* --free-full : Estimate both reference bias parameters and the contamination parameters using sequence+array method
* --ignoreRG : ignore the read grouup level comparison and compare samples only (recommended for an expedited run)
* --ignoreOverlapPair : ignore overlapping pair end fragment covering the same base. Disabling this option may decrease the sensitivity of the method when the insert size is short (with slight gain in the computational speed)
* --noEOF : do not check the EOF marker of the BAM file (for earlier version of BAM)
* --precise : calculate the likelihood in log-scale for high-depth data (recommended when --maxDepth is greater than 20. Can be a little bit slower)
* --minMapQ : minimum mapping quality of the sequence reads to compare
* --minQ : minimum base quality to include
* --maxQ : maximum base quality to cap
* --grid : the grid interval to search the optimum before running Brent's algorithm.
* --refRef : Initial Pr(refBase|HOMREFGeno) parameter
* --refHet : Initial Pr(refBase|HETGeno) parameter
* --refAlt : Initial Pr(refBase|HOMALTGeno) parameter
* --out : output file prefix (required)
* --verbose : print the progress of the method on the screeen
{{PhoneHomeParameters|hdr=====|bullet=1}}

== Principle of Operation ==

Each read group in a BAM file is evaluated independently. This means that in file with multiple read groups, problems will be flagged at the read group level (a plus). However, it also means that it might be hard to discern the correct assignment of read groups with very little data.

For each aligned base that overlaps a known genotype, we calculate the probability the probability that it was derived from a particular known genotype. This comparison considers only bases that overlap previously known genotypes and that meet the base quality and mapping quality thresholds.

Each individual in a pedigree has a different combination of genotypes, and bamGenotypeCheck will systematically search for the individual whose genotypes best match the observed read data.

For more about the technical details, see the page [[Verifying Sample Identities - Implementation]]

== Reference ==

Please cite the following paper:

G. Jun, M. Flickinger, K. N. Hetrick, Kurt, J. M. Romm, K. F. Doheny, G. Abecasis, M. Boehnke,and H. M. Kang, ''Detecting and Estimating Contamination of Human DNA Samples in Sequencing and Array-Based Genotype Data'', American journal of human genetics doi:10.1016/j.ajhg.2012.09.004 (volume 91 issue 5 pp.839 - 848)

== Contamination in Array Data ==

[[VerifyIDintensity]] or [[BAFRegress]] can estimate sample contamination from Illumina genotype array data.

== Acknowledgements ==

VerifyBamID is a result from collaborative effort by Goo Jun, Matthew Flickinger, Mary Kate Wing, Goncalo Abecasis, Michael Boehnke, and Hyun Min Kang. Please email to Hyun Min Kang [[mailto:hmkang@umich.edu| hmkang@umich.edu]] for any questions.

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:26:28Z

Hmkang: /* Extracting only exonic SNPs */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/
export REF=/net/seqshop-server/home/mktrost/seqshop/singleSample/ref/gotcloud.ref
export GC=~/seqshop/gotcloud

export SAMPLE=SampleXX
export OUT=~/$SAMPLE/output

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz
$HK/tabix -pvcf $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $REF/hs37d5.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:26:17Z

Hmkang: /* Extracting only exonic SNPs */

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:23:30Z

Hmkang: /* Friday : More SNP Analysis */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/
export REF=/net/seqshop-server/home/mktrost/seqshop/singleSample/ref/gotcloud.ref
export GC=~/seqshop/gotcloud

export SAMPLE=SampleXX
export OUT=~/$SAMPLE/output

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $REF/hs37d5.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:22:04Z

Hmkang: /* Annotation / Lookup against dbSNP */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/
export REF=/net/seqshop-server/home/mktrost/seqshop/singleSample/ref/gotcloud.ref

export SAMPLE=SampleXX
export OUT=~/$SAMPLE/output

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $REF/hs37d5.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:21:49Z

Hmkang: /* Annotating your genome */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/
export REF=/net/seqshop-server/home/mktrost/seqshop/singleSample/ref/gotcloud.ref

export SAMPLE=SampleXX
export OUT=~/$SAMPLE/output

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid --autosomes

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $REF/hs37d5.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:21:32Z

Hmkang: /* Extracting only exonic SNPs */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/
export REF=/net/seqshop-server/home/mktrost/seqshop/singleSample/ref/gotcloud.ref

export SAMPLE=SampleXX
export OUT=~/$SAMPLE/output

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid --autosomes

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno --autosomes

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $REF/hs37d5.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:20:59Z

Hmkang: /* Annotation / Lookup against dbSNP */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/
export REF=/net/seqshop-server/home/mktrost/seqshop/singleSample/ref/gotcloud.ref

export SAMPLE=SampleXX
export OUT=~/$SAMPLE/output

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid --autosomes

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno --autosomes

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome --autosomes

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $REF/hs37d5.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:20:40Z

Hmkang: /* Extracting only exonic SNPs */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/
export REF=/net/seqshop-server/home/mktrost/seqshop/singleSample/ref/gotcloud.ref

export SAMPLE=SampleXX
export OUT=~/$SAMPLE/output

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno --autosomes

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome --autosomes

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $REF/hs37d5.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:20:15Z

Hmkang: /* Annotating your genome */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/
export REF=/net/seqshop-server/home/mktrost/seqshop/singleSample/ref/gotcloud.ref

export SAMPLE=SampleXX
export OUT=~/$SAMPLE/output

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno --autosomes

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $REF/hs37d5.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:04:22Z

Hmkang: /* Friday : More SNP Analysis */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/
export REF=/net/seqshop-server/home/mktrost/seqshop/singleSample/ref/gotcloud.ref

export SAMPLE=SampleXX
export OUT=~/$SAMPLE/output

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $REF/hs37d5.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T15:03:06Z

Hmkang: /* Friday : More SNP Analysis */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/
export REF=/net/seqshop-server/home/mktrost/seqshop/singleSample/ref/gotcloud.ref

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $REF/hs37d5.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T14:53:22Z

Hmkang: /* Extracting only exonic SNPs */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-make --repeat-chr --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6 --out runmake.exome

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T14:52:45Z

Hmkang: /* Annotating your genome */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6 --out runmake.anno

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T14:52:14Z

Hmkang: /* Annotation / Lookup against dbSNP */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,12,13,22 $HK/../data/gwascatalog/gwascatalog.txt | grep -w rs17766217

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T14:50:05Z

Hmkang: /* Annotation / Lookup against dbSNP */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example, rs17766217) -- How can we find its position?
$HK/tabix ~/NA12878/output/vcfs/chr8/chr8.filtered.rsid.vcf.gz 8:128504497 | less
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

== OVERALL COURSE FEEDBACK! ==
Please provide feedback:
https://docs.google.com/forms/d/1pxfPXKwWfA71ZJM99Sevs3MwAUz2UbHAR8dnRI-kRNM/viewform

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T14:25:49Z

Hmkang: /* Annotation / Lookup against dbSNP */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6 --out runmake.rsid

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T14:24:28Z

Hmkang: /* Annotation / Lookup against dbSNP */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T14:23:07Z

Hmkang: /* Annotation / Lookup against dbSNP */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== Reviewing Indel Results ===
Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===
In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin
export EPACTS=/net/seqshop-server/home/mktrost/seqshop/epacts/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbsnp_142.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T13:35:37Z

Hmkang: /* Environmental Variables */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content mw-collapsed">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Association Analysis Tutorial ===
Now we are going to run the Association Analysis Practical

Please go to: [[SeqShop: Association Analysis, May 2015]]

We will look at our own genomes again after the practical.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Reviewing Indel Results ===
Set these values. Also, be sure to specify your sample name (or NA12878) instead of SampleXX
export SAMPLE=SampleXX
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===

==== Environmental Variables ====

If you didn't set the environmental variable, you can set it again

export SAMPLE=SampleXX (MAKE SURE TO CHANGE XX to your number or use NA12878 instead)
export OUT=~/$SAMPLE/output

In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T13:26:14Z

Hmkang: /* Annotating your genome */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content mw-collapsed">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Association Analysis Tutorial ===
Now we are going to run the Association Analysis Practical

Please go to: [[SeqShop: Association Analysis, May 2015]]

We will look at our own genomes again after the practical.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Reviewing Indel Results ===
Set these values. Also, be sure to specify your sample name (or NA12878) instead of SampleXX
export SAMPLE=SampleXX
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===

==== Environmental Variables ====

If you didn't set the environmental variable, you can set it again

source /net/seqshop-server/home/mktrost/seqshop/setup.txt
export SAMPLE=SampleXX (MAKE SURE TO CHANGE XX to your number or use NA12878 instead)
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --repeat-chr --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T13:25:40Z

Hmkang: /* Annotation / Lookup against dbSNP */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content mw-collapsed">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Association Analysis Tutorial ===
Now we are going to run the Association Analysis Practical

Please go to: [[SeqShop: Association Analysis, May 2015]]

We will look at our own genomes again after the practical.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Reviewing Indel Results ===
Set these values. Also, be sure to specify your sample name (or NA12878) instead of SampleXX
export SAMPLE=SampleXX
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===

==== Environmental Variables ====

If you didn't set the environmental variable, you can set it again

source /net/seqshop-server/home/mktrost/seqshop/setup.txt
export SAMPLE=SampleXX (MAKE SURE TO CHANGE XX to your number or use NA12878 instead)
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --repeat-chr --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-command-wgs --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T13:25:09Z

Hmkang: /* Environmental Variables */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content mw-collapsed">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Association Analysis Tutorial ===
Now we are going to run the Association Analysis Practical

Please go to: [[SeqShop: Association Analysis, May 2015]]

We will look at our own genomes again after the practical.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Reviewing Indel Results ===
Set these values. Also, be sure to specify your sample name (or NA12878) instead of SampleXX
export SAMPLE=SampleXX
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===

==== Environmental Variables ====

If you didn't set the environmental variable, you can set it again

source /net/seqshop-server/home/mktrost/seqshop/setup.txt
export SAMPLE=SampleXX (MAKE SURE TO CHANGE XX to your number or use NA12878 instead)
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/apigenome/bin

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-command-wgs --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-command-wgs --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T13:23:46Z

Hmkang: Reverted edits by Hmkang (talk) to last revision by Mktrost

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content mw-collapsed">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Association Analysis Tutorial ===
Now we are going to run the Association Analysis Practical

Please go to: [[SeqShop: Association Analysis, May 2015]]

We will look at our own genomes again after the practical.

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Reviewing Indel Results ===
Set these values. Also, be sure to specify your sample name (or NA12878) instead of SampleXX
export SAMPLE=SampleXX
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===

==== Environmental Variables ====

If you didn't set the environmental variable, you can set it again

source /net/seqshop-server/home/mktrost/seqshop/setup.txt
export SAMPLE=SampleXX (MAKE SURE TO CHANGE XX to your number or use NA12878 instead)
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/net/seqshop-server/home/hmkang/seqshop/bin

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-command-wgs --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-command-wgs --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T13:22:22Z

Hmkang: /* Friday */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:1000px">

== Friday ==
<div class="mw-collapsible">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Reviewing Indel Results ===
Set these values. Also, be sure to specify your sample name (or NA12878) instead of SampleXX
export SAMPLE=SampleXX
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===

==== Environmental Variables ====

If you didn't set the environmental variable, you can set it again

source /net/seqshop-server/home/mktrost/seqshop/setup.txt
export SAMPLE=SampleXX (MAKE SURE TO CHANGE XX to your number or use NA12878 instead)
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/home/hmkang/apigenome/bin/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --repeat-chr --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --repeat-chr --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T13:20:52Z

Hmkang:

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Reviewing Indel Results ===
Set these values. Also, be sure to specify your sample name (or NA12878) instead of SampleXX
export SAMPLE=SampleXX
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===

==== Environmental Variables ====

If you didn't set the environmental variable, you can set it again

source /net/seqshop-server/home/mktrost/seqshop/setup.txt
export SAMPLE=SampleXX (MAKE SURE TO CHANGE XX to your number or use NA12878 instead)
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/home/hmkang/apigenome/bin/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-make --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --repeat-chr --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-make --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --repeat-chr --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T13:18:34Z

Hmkang: /* Friday */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Reviewing Indel Results ===
Set these values. Also, be sure to specify your sample name (or NA12878) instead of SampleXX
export SAMPLE=SampleXX
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===

==== Environmental Variables ====

If you didn't set the environmental variable, you can set it again

source /net/seqshop-server/home/mktrost/seqshop/setup.txt
export SAMPLE=SampleXX (MAKE SURE TO CHANGE XX to your number or use NA12878 instead)
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/home/hmkang/apigenome/bin/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-command-wgs --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-command-wgs --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T13:17:48Z

Hmkang: /* Friday */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content mw-expanded">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Reviewing Indel Results ===
Set these values. Also, be sure to specify your sample name (or NA12878) instead of SampleXX
export SAMPLE=SampleXX
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===

==== Environmental Variables ====

If you didn't set the environmental variable, you can set it again

source /net/seqshop-server/home/mktrost/seqshop/setup.txt
export SAMPLE=SampleXX (MAKE SURE TO CHANGE XX to your number or use NA12878 instead)
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/home/hmkang/apigenome/bin/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-command-wgs --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-command-wgs --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>

SeqShop: Calling Your Own Genome, May 2015

2015-05-22T13:17:27Z

Hmkang: /* Friday */

__TOC__
<div class="mw-collapsible" style="width:500px">
''Login instructions for seqshop-server''
<div class="mw-collapsible-content">
{{SeqShopLogin}}
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:500px">
== Tuesday - Start SNP Calling ==
<div class="mw-collapsible-content">
=== Setup Screen ===
The snpcall pipeline will run overnight, but you'll want to log out.
; How do I leave something running on the server even if I log out?
: One solution is screen!

; How do I use screen?
: Before running your command, you need to start screen:
: <pre>screen</pre>

[[File:Screen.png]]

As it says, press <code>Space</code> or <code>Return</code>.
* It should now look basically the same as your normal command line.

=== Setup Variables ===
Set these values. Also, be sure to specify your sample name instead of SampleXX
export SAMPLE=SampleXX
or
export SAMPLE=NA12878

Point to your GotCloud & your output directory:
export GC=~/seqshop/gotcloud
export OUT=~/$SAMPLE/output

=== List of BAMs ===
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

=== Configuring SNPCALL ===

cat ~/$SAMPLE/gotcloud.conf

You will see this:
<pre>
# References
SS_DIR = /net/seqshop-server/home/mktrost/seqshop/singleSample
REF_DIR = $(SS_DIR)/ref/gotcloud.ref/

######### ALIGNMENT ########
MAP_TYPE = BWA_MEM
FASTQ_LIST = fastq.list
BATCH_TYPE =
BATCH_OPTS =
BWA_THREADS = -t 6

# SNP Call Settings
UNIT_CHUNK = 20000000 # Chunk size of SNP calling : 20Mb
VCF_EXTRACT = $(SS_DIR)/snpOnly.vcf.gz
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE

EXT_DIR = $(SS_DIR)/ext
EXT = $(EXT_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(EXT_DIR)/chrCHR.filtered.sites.vcf.gz
</pre>

=== Running SNP Calling ===
Run GotCloud snpcall with 8 jobs running in parallel
* Why 8?
** You want to run as many as you can.
** 2-3 of you on the machine - 3*8 = 24 jobs will be running in parallel on that machine
${GC}/gotcloud snpcall --conf $SAMPLE/gotcloud.conf --numjobs 8 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

=== Log Out ===
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

If you have not detached from screen:
Ctrl-a d

exit PuTTY

=== FEEDBACK! ===
Please provide feedback on today:

https://docs.google.com/forms/d/1ADTkBjzT-QNj2lrejyqGqDaahTponrw20kSgDNwqwH4/viewform

</div>
</div>

<div class="mw-collapsible" style="width:500px">

== Thursday ==
<div class="mw-collapsible-content">

=== Checking if snpcall Completed ===
==== Resume screen to Check Jobs ====

;How do you log back into screen?
screen -r
This will resume an already running screen.

Your screen session still has your environment variables set, so you do not need to reset them.

Verify you got a "completed successfully" message.

How long did snpcall calling take? Look at the log message - time in seconds.

==== List of BAMs ====
The list of BAMs has already been created (just 1 BAM, your sample).
* But it is simply SAMPLE\tBAM_name, so easy to figure out

cat ~/$SAMPLE/output/bam.list

:<code>SampleXX SampleXX/output/bams/SampleXX.recal.bam</code>
* Relative path, so assumes running from your home directory (I prefer absolute paths, but for simplicity of the workshop, we just use relative path).

==== GotCloud INDEL Configuration ====

cat ~/$SAMPLE/gotcloud.conf

Same as it looked the other day with no special Configuration settings for INDEL calling.

==== Running INDEL ====
Run GotCloud indel with 6 jobs running in parallel
${GC}/gotcloud indel --conf $SAMPLE/gotcloud.conf --numjobs 6 --outdir $OUT
* Only need the configuration, number of threads, and the output directory, rest is specified within the configuration.

This will run overnight. We will check if it completed at the practical in the morning.

==== Log Out ====
;Want to log out and leave your job running?
In the screen window, type:
Ctrl-a d
(Hold down Ctrl and type 'a', let go of both and type 'd')
* This will "detach" from your screen session while your alignment continues to run.

exit PuTTY

=== FEEDBACK!===
Please provide feedback for today.
https://docs.google.com/a/umich.edu/forms/d/1iES6usHxLB7Ec9hRxtqYgH7v05lU3Ume4VJcksx8Ogg/viewform

</div>
</div>
</div>
</div>

<div class="mw-collapsible mw-collapsed" style="width:1000px">

== Friday ==
<div class="mw-collapsible-content">

=== SeqShop: Ancestry On Your Own Genome, May 2015 ===
[[SeqShop: Ancestry On Your Own Genome, May 2015]]

=== Return to SeqShop: Ancestry On Your Own Genome, May 2015 ===
Return to [[SeqShop:_Ancestry_On_Your_Own_Genome,_May_2015#Checking_if_Pileup_finished]]

=== Reviewing Indel Results ===
Set these values. Also, be sure to specify your sample name (or NA12878) instead of SampleXX
export SAMPLE=SampleXX
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

Look in the output directory
ls ~/$SAMPLE/output

What in that directory was produced by indel calling?
* <code>gotcloud.indel.conf</code>
** dump of all configuration settings for this run
* <code>gotcloud.indel.Makefile</code>
** Makefile that was generated to manage all of the commands to be run
* <code>gotcloud.indel.Makefile.log</code>
** log of all commands run by the Makefile
* <code>indel/</code>
** indel output directory

Let's look at the indel output
ls ~/$SAMPLE/output/indel
* 3 directories
** aux - intermediate files
** indelvcf - intermediate files
** '''final indel files'''

Final indel directory:
ls ~/$SAMPLE/output/indel/final
* '''all.genotypes.vcf.gz - output VCF'''
* '''all.genotypes.vcf.gz.tbi - output VCF index file to allow jumping to positions'''
* all.genotypes.vcf.gz.tbi.OK - completion indicator
* merge/ - directory with per chromosome bcf (binary vcf) files
* all.genotypes.vcf.gz.OK - completion indicator
* all.genotypes.vcf.gz.tbi.log - log
* concat.log - log

==== Looking at final INDEL VCF ====

Note that because this is a single sample calling, many of the INFO fields are less meaningful as many of the values like HWE p values, allele frequencies, inbreeding coefficient are a function of a population.
Nonetheless, we may examine the results. First, we see how many indels were discovered for your genome:

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz

no. Indels : 588566
2 alleles (ins/del) : 588566 (0.87) [273261/315305]

This gives use 588,566 indels with an insertion deletion ratio of 0.87.

We next look at the filtered set. The PASS filter reduces the setof indels to a non overlapping set and the INFO.AC!=0 extracts all indels that are either heterozygous or homozygous alternative.
Some indels that were originally discovered were found to be the homozygous reference genotype. Invariably, these are relative high depth calls where the
alternative allele is discovered less or is mis-specified.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC!=0"

no. Indels : 549963
2 alleles (ins/del) : 549963 (0.91) [261480/288483]

About 38K indels were removed, the insertion deletion ratio increases to 0.91. Note that in general, for high depth data, discovered indels are reported with insertion deletion ratios
close to 1. So this is a good sign. Next generation sequencing errors are bias for deletions.

It is possible to perform a slightly more stringent filtering using allele balance. The allele balance estimator in this case is meaningful still for an individual because it is a function of read depth.
Note that AB>0.5 denotes reference bias and AB<0.5 denotes alternative allele bias.

$GC/bin/vt peek ~/$SAMPLE/output/indel/final/all.genotypes.vcf.gz -f "PASS&&INFO.AC>0&&INFO.AB<0.7&&INFO.AB>0.3"

no. Indels : 490965
2 alleles (ins/del) : 490965 (0.92) [235254/255711]

The insertion deletion ratio increases from 0.91 to 0.92.

=== Friday: Reviewing SNPCALL Results ===
Look in the output directory
ls ~/$SAMPLE/output

Look at the vcfs:
ls ~/$SAMPLE/output/vcfs

=== Friday : More SNP Analysis ===

==== Environmental Variables ====

If you didn't set the environmental variable, you can set it again

source /net/seqshop-server/home/mktrost/seqshop/setup.txt
export SAMPLE=SampleXX (MAKE SURE TO CHANGE XX to your number or use NA12878 instead)
source /net/seqshop-server/home/mktrost/seqshop/setupSS.txt

In addition, set another environmental variable for locating the binaries for custom analysis

export HK=/home/hmkang/apigenome/bin/

==== Annotation / Lookup against dbSNP ====

If you want to add rsIDs to your variant files, you can do this by running the following command

$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz

If you want to run this command across all chromosomes in parallel, you can use the special script run-command-wgs

$HK/run-command-wgs --cmd "$HK/vcf-add-rsid -vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --db $HK/../data/dbSNP.b138/dbsnp_138.b37.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz" --numjobs 6

Looking up SNPs by rsID is possible by (for example)
$HK/vcf-lookup-rsid --vcf $OUT/vcfs/chr1/chr1.filtered.vcf.gz --sepchr --rs rs17766217
* Be sure to look at the QUAL & your sample's PL, and not just the GL field. Check if QUAL is 0 or PL is 0,0,0 - NS is also probably 0; DP is probably 0. That means you probably didn't have any copies, so your GT may not be correct/is unknown.

If you want to browse the rsIDs of known GWAS SNPs, you can do this by
cut -f 1,8,22 $HK/../data/gwascatalog/gwascatalog.txt | less

==== Annotating your genome ====

You can annotate your genome using EPACTS software packages. Individual chromosome can be annotated by running.
$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz

Or you can run multiple chromosomes in parallel in one command
$HK/run-command-wgs --cmd "$EPACTS/bin/epacts anno --in $OUT/vcfs/chr1/chr1.filtered.rsid.vcf.gz --out $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz" --numjobs 6

==== Extracting only exonic SNPs ====

If you want to look at the exonic SNPs, you can extract using the following command
$HK/run-command-wgs --cmd "($HK/tabix -H $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz; zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.vcf.gz | grep Exon;)| $HK/bgzip -c > $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz" --numjobs 6

And they can be combined as follows
(zcat $OUT/vcfs/chr1/chr1.filtered.rsid.anno.exon.vcf.gz; zcat $OUT/vcfs/chr[2-9]/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chr??/chr*.filtered.rsid.anno.exon.vcf.gz $OUT/vcfs/chrX/chrX.filtered.rsid.anno.exon.vcf.gz | grep -v ^#) | $HK/bgzip -c > $OUT/wgs.filtered.rsid.anno.exon.vcf.gz

==== Exonic Variants NOT found by 1000G ====

If you are interested in rare variants that are not identified by 1000G, you can extract them by running
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | less

For example,
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

will give you the counts of variants, separate by the filtering results

* Q1. How manny novel silent, missense, and nonsense SNPs are found? Is that too few, too small, or just about right?
* Q2. Looking at each functional category, which functional categories has largest fraction of SNPs failed filter? Why do you think it is?
* Q3. Can you exclude the sites that are also in dbSNP, and count how many nonsense variants are left?

To also exclude those in dbsnp:
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" | grep -v rs| perl -lane 'print "$1\t$F[6]" if ( /ANNO=([^;:]+)/)' | sort | uniq -c

Exclude dbsnp and look at Stop_Gain variants
zcat $OUT/wgs.filtered.rsid.anno.exon.vcf.gz | grep "EXTFILTER=NA,NA" | grep -v -w "0/0" |grep -v rs | perl -lane 'print "$_" if ( /ANNO=Stop_Gain/)' |grep -w PASS

Want to see this from the BAM file? Use samtools tview:
$GC/bin/samtools tview $SAMPLE/output/bams/$SAMPLE.recal.bam $GC/gotcloud.ref/human.g1k.v37.fa
Use 'g' & enter the Chr:Pos
* Some patterns may indicate not real variants.

If you want to know predicted functional significance of a particular variant, you can search by

$HK/tabix $HK/../data/CADD/whole_genome_SNVs.tsv.gz [chr]:[pos] | head -3

The phred score at the last column quantifies the degree of functional significance

</div>
</div>