Genome Analysis Wiki - User contributions [en]

Help:Contents

2017-03-01T20:36:12Z

Pjvh:

If you have any questions about formatting or capabilities of the wiki software, visit [[http://www.mediawiki.org/wiki/Help:Contents MediaWiki Help]] for a user's guide.

If you are looking for a person to contact about a specific page on this wiki,
# Go to the page on this wiki that you're interested in.
# Click the tab '''history''' at the top of that page.
# Each row on that page is an edit someone made. Each shows the time and date, username, size of the page, and size of the edit (like '''(+395)'''). Look for someone who's made some large recent changes.
# Search Google for ''site:umich.edu'' followed by their username.

Minimac

2017-03-01T19:29:49Z

Pjvh:

'''You may want to learn about new and improved [[Minimac3]].'''

'''minimac''' is a low memory, computationally efficient implementation of the MaCH algorithm for genotype imputation. It is designed to work on phased genotypes and can handle very large reference panels with hundreds or thousands of haplotypes. The name has two parts. The first, "mini", refers to the modest amount of computational resources it requires. The second, "mac", is short hand for [[MaCH]], our widely used algorithm for genotype imputation.

There are several minimac related pages on this wiki. The major ones are:

* [[Minimac]] - This page, the main minimac page.
* [[Minimac: Tutorial]] - A short minimac tutorial.
* [[Minimac: 1000 Genomes Imputation Cookbook]] - Cookbook instructions for 1000 Genomes Imputation with Minimac
* [[Minimac Command Reference]] - Summary of minimac options
* [[Minimac Diagnostics]] - Summary of diagnostics for imputation performance generated by minimac

*[https://imputationserver.sph.umich.edu Imputation server] - We are running imputation (and pre-phasing) for you!

= Download =
A binary Linux (64 bit) version of minimac is available [http://csg.sph.umich.edu/cfuchsb/minimac-beta-2013.7.17.tgz from here] and source code [http://csg.sph.umich.edu/cfuchsb/minimac.src.tgz from here]

The current version of minimac should be stamped 2013.7.17 - if your version shows a different version number or date stamp when it runs, it is not current.

If you use this beta version, please be sure to stop by the [http://www.sph.umich.edu/csg/abecasis/MaCH/download/ MaCH download page] and fill out the registration form, so that we can let you know when an official release is available and keep you updated with respect to any bug fixes.

== Multiprocessor Version ==

The current version of minimac comes in two flavours, <code>minimac</code> and <code>minimac-omp</code>. The latter version uses the [[OpenMP]] protocol to support multi-threading, resulting in faster throughput.

BE AWARE: since this version of minimac runs in parallel the order of samples in the output files (*dose, *haps,...) will vary between runs. Therefore, e.g. chunks have to be merged by sample id.

== Change log ==

2013.7.17

- minor bug fixes

-- all variants (SNPs, InDels, SVs) in the reference VCF will be imputed - independent from the FILTER column setting

- improved performance (Thanks to David Hinds - see also [[minimac2]] for the full set of performance improvements)

2012.11.16

- minor bug fixes

2012.10.9

- added: improved support for [http://www.shapeit.fr ShapeIT] phased haplotypes

2012.10.3

- added: full support for reference panel based chunking

2012.9.22

- fixed: chunk chromosome bug

2012.8.6 (early adopter)

- added: chromosome X support

2012.3.14

- fixed: problem with --startposition

2012.2.29

- added: VCF support

- added: IDR (Insertion, Deletion, Reference) support

== Questions and Comments ==

Please contact [mailto:goncalo@umich.edu Goncalo Abecasis] or [mailto:cfuchsb@umich.edu Christian Fuchsberger].

= Performance =

== Pre-phasing ==
For the pre-phasing step, cost increases quadratically with the number of states and linearly with the number of rounds. The following table provides a simple example.

{| class="wikitable" border="1" cellpadding="2"
|- bgcolor="lightgray"
! States
! Cost per round
|-
| 100 states
| 3 min
|-
| 200 states
| 12 min = (3 min * 22)
|-
| 400 states
| 48 min = (3 min * 42)
|-
| 500 states
| 75 min = (3 min * 52)
|}

So, in this case running haplotyping with 500 states and 10 rounds would require 75 min * 10 = 750 min.

Typically, haplotype quality improves rapidly with the number of states but only slowly with the number of rounds. We recommend running ~20 rounds of the MaCH haplotyper and selecting a number of states as high as your patience will allow (but ideally greater than 200).

== Imputation ==

A good rule of thumb is that minimac should take about 1 hour to impute 1,000,000 markers in 1,000 individuals using a reference panel with 100 haplotypes. Performance should scale linearly with respect to all these factors. So, your approximate computing time in hours should be about:

:<math>
E(\mbox{Run Time in Hours}) = N_{markers} * N_{individuals} * N_{haplotypes} * 10^{-11}
</math>

These statistics refer to a single core in a modern Intel CPU core and, although your mileage will vary, most modern CPUs should be no more than a few times faster (or slower) than that.

If you are estimating model parameters at the same time as imputing missing genotypes, you can account for the time needed for parameter estimation with the following formula:

:<math>
E(\mbox{Run Time in Hours}) = N_{markers} * ({N_{individuals} + N_{rounds} * N_{states} * 0.75 }) * N_{haplotypes} * 10^{-11}
</math>

In this updated formula, Nrounds represents the number of iterations used for parameter refinement and Nstates represents the maximum number of reference and target haplotypes considered for each update.

= Getting Started =

Using minimac for genotype imputation involves two steps. First, you will have to estimate haplotypes for your entire sample -- this will be the more computationally demanding step. Once that is done, you will be ready to quickly impute missing genotypes using the reference panel of your choice.

== Estimating Haplotypes for Your Sample ==

For the haplotyping step, we current recommend using [[MaCH]] with the --phase command line option. As input [[MaCH]] will need [[Merlin]] format pedigree and data files. All markers should be ordered according to their physical position and alleles should be labeled on the forward strand.

=== Preparing Your Data ===

To get started, you will need to store your data in [[Merlin]] format pedigree and data files, one per chromosome. For details, of the Merlin file format, see the [http://csg.sph.umich.edu/abecasis/Merlin/tour/input_files.html Merlin Tutorial].

Within each file, markers should be stored by chromosome position. Alleles should be stored in the forward strand and can be encoded as 'A', 'C', 'G' or 'T' (there is no need to use numeric identifiers for each allele).

We recommend that, if at all possible, you should phase your chromosomes according to NCBI build 37. Future releases of the 1000 Genomes Reference panel and other public sets of reference haplotypes are expected to use this genome build.

If figuring out position and strand for each marker seems like hard work, don't despair. For you, this should be the hardest bit of the entire process! For the computer, the fun is about to start.

==== NCBI build 36 / NCBI build 37 ====
The 1000G June reference panel is on build 36, future 1000G reference panels will be on build 37. This has also some impact on the pre-phasing:
some SNPs will ordered differently in each genome build (we expect order will be more accurate in more recent builds!). Purists will claim that imputation using NCBI build 36 and NCBI build 37 reference panels requires phasing using the exact same reference panel -- others will claim this makes little difference.

=== Running MaCH ===

A typical MaCH command line to estimate phased haplotypes might look like this:

mach1 -d sample.dat -p sample.ped --rounds 20 --states 200 --phase --interim 5 --sample 5 --compact

This will request that MaCH estimate haplotypes for your sample, using 20 iterations of its Markov sampler and conditioning each update on up to 200 haplotypes. A summary description of these parameters follows (but for a more complete description, you should go to the MaCH website):

{| class="wikitable" border="1" cellpadding="2"
|- bgcolor="lightgray"
! Parameter
! Description
|-
|style=white-space:nowrap|<code>-d sample.dat</code>
| Data file in [http://csg.sph.umich.edu/abecasis/Merlin/tour/input_files.html Merlin format]. Markers should be listed according to their order along the chromosome.
|-
| <code>-p sample.ped</code>
| Pedigree file in [http://csg.sph.umich.edu/abecasis/Merlin/tour/input_files.html Merlin format]. Alleles should be labeled on the forward strand.
|-
| <code>--states 200</code>
| Number of haplotypes to consider during each update. Increasing this value will typically lead to better haplotypes, but can dramatically increase computing time and memory use. A value of 200 - 400 is typical.
|-
| <code>--rounds 20</code>
| Iterations of the Markov sampler to use for haplotyping. Typically, using 20 - 30 rounds should give good results. To obtain better results, it is usually better to increase the <code>--states</code> parameter.
|-
| <code>--interim 5</code>
| Request that intermediate results should be saved to disk periodically. These will facilitate analyses in case a run doesn't complete.
|-
| <code>--sample 5</code>
| Request that random (but plausible) sets of haplotypes for each individual should be drawn every 5 iterations. This parameter is optional, but for some rare variant analyses, these alternative haplotypes can be very useful.
|-
| <code>--phase</code>
| Tell [[MaCH]] to estimate phased haplotypes for each individual.
|-
| <code>--compact</code>
| Reduce memory use at the cost of approximately doubling runtime. This option is recommended for most GWAS scale datasets and computing platforms.
|}

== Imputation into Phased Haplotypes ==

Imputing genotypes using '''minimac''' is an easy and straightforward process: after selecting a set of reference haplotypes, plugging-in the target haplotypes from the previous step and setting the number of rounds to use for the model parameter estimation, imputation should proceed rapidly.

=== Creating SNP List File ===

'''Minimac''' requires a file listing markers in the haplotype file. This file can be easily generated by extracting the second column from the .dat file. In a standard Unix system, a command like this should do:

cut -f 2 -d " " sample.dat > target.snps

=== Running Minimac ===

A typical minimac command line might look like this:

==== using a VCF reference panel ====
minimac --vcfReference --refHaps ref.vcf.gz --haps target.hap.gz --snps target.snps.gz --rounds 5 --states 200 --prefix results

'''Note''': GWAS SNPs (file --snps target.snps.gz) are by default expected to be in the chr:pos format e.g. 1:1000 and on build37/hg19; otherwise, please set the --rs flag

==== using a MaCH reference panel ====

minimac --refHaps ref.hap.gz --refSnps ref.snps.gz --haps target.hap.gz --snps target.snps.gz --rounds 5 --states 200 --prefix results

A detailed description of all minimac options is available [[Minimac Command Reference|elsewhere]]. Here is a brief description of the above parameters:

{| class="wikitable" border="1" cellpadding="2"
|- bgcolor="lightgray"
! Parameter
! Description
|-
| <code>--refSnps ref.snps.gz </code>
| List of SNPs in the reference panel
|-
| <code>--refHaps ref.hap.gz </code>
| Reference haplotypes (e.g. from [http://csg.sph.umich.edu/abecasis/MACH/download/ MaCH download page])
|-
| <code>--vcfReference </code>
| This option specifies that the provided --refHaps file is provided in VCF format , no --refSNPs file needed.
|-
| <code>--rs </code>
| In combination with --vcfReference, allows to use rs GWAS SNP identifiers
|-
| <code>--snps target.snps.gz </code>
| SNPs in phased haplotypes. These should largely be a subset of the SNPs in the reference panel.
|-
| <code>--haps target.hap.gz </code>
| Phased haplotypes where missing genotypes will be imputed.
|-
| <code>--sample target.sample </code>
| Sample list in ShapeIT format.
|-
| <code>--shape_haps target.hap.gz </code>
| ShapeIT phased haplotypes where missing genotypes will be imputed.
|-
| <code>--chr 22</code>
| Chromosome for which we will carry out imputation (needed to run ShapeIT with chr:pos identifiers - default setting).
|-
| <code>--rounds 5</code>
| Rounds of optimization for model parameters, which describe population recombination rates and per SNP error rates.
|-
| <code>--states 200</code>
| Maximum number of reference (or target) haplotypes to be examined during parameter optimization.
|-
| <code>--prefix imputed</code>
| Optionally, a string that is used to help generate output file names.
|}

=== Reference Haplotypes ===

Reference haplotypes generated by the 1000 Genomes project and formatted so that they are ready for analysis are available from the [http://csg.sph.umich.edu/abecasis/MACH/download/ MaCH download page]. As of this writing, the most recent set of haplotypes are based on genotype calls were generated in May 2011 and are an interim analysis of Project's Phase I data.

=== Imputation quality evaluation ===
To evaluate imputation quality, Minimac hides data for each genotyped SNP in turn and calculates 3 statistics:
* looRSQ - this is the estimated rsq for that SNP (as if SNP weren't typed).
* empR - this is the empirical correlation between true and imputed genotypes for the SNP. If this is negative, the SNP is probably flipped.
* empRSQ - this is the actual R2 value, comparing imputed and true genotypes.

These statistics can be found in the .info file

=== Additional Sources of Information ===

If the combination of MaCH and Minimac still runs too slowly for you, and you have access to a multi-processor compute cluster, you can look at [[ChunkChromosome]] page to learn how to conveniently split each chromosome into multiple segments that can be analyzed in parallel. For information on how to put the resulting chunks back together, see [[Ligate Minimac|this page]].

If you are especially interested in 1000 Genomes Imputation, then you should look at the [[Minimac: 1000 Genomes Imputation Cookbook]].

== X Chromosome Imputation ==
minimac supports the imputation of genotypes on the X chromosome (non-pseudo-autosomal part).

# Split the X chromosome pedigree file by sex.
## For females: follow the same protocol as for autosomes (phase with MaCH and impute with minimac).
## For males
### Remove the pseudo-autosomal part (for build hg18: chrX:1-2709520 and chrX:154584238-154913754 ; for build hg19 chrX:60001-2699520 and chrX:154931044-155260560)
### Convert the pedigree file into a MaCH haplotype file (missing genotypes should be encoded as: "0" or "." or "N" ) and impute using minimac as described above.

:::: '''<Example of a male only pedigree file>'''
:::: FAM1003 ID1234 0 0 M A/A A/A C/C
:::: FAM1004 ID5678 0 0 M 0/0 C/0 G/G
:::: ...
:::: '''<End of pedigree file>'''

:::: ''Note that, consistent with the Merlin convention, hemizygous males are listed as if they were homozygous.''

:::: '''<Example of the corresponding haplotype file>'''
:::: FAM1003->ID1234 HAPLO1 AAC
:::: FAM1003->ID1234 HAPLO2 AAC
:::: FAM1004->ID5678 HAPLO1 0CG
:::: FAM1004->ID5678 HAPLO2 0CG
:::: ...
:::: '''<End of the corresponding haplotype file>'''

= Post-imputation Association Analysis =
== Quantitative Traits ==
Please use [http://csg.sph.umich.edu/yli/mach/download/mach2qtl.source.V108.tgz mach2qtl].

== Binary Traits ==
Please use [http://csg.sph.umich.edu/yli/mach/download/mach2dat.source.1.0.18.tgz mach2dat]. Versions 1.0.18 and above accommodate to minimac output.

= Reference =

If you use [[minimac]] or [[minimac2]] please cite:

Fuchsberger C, Abecasis GR, Hinds DA. minimac2: faster genotype imputation. Bioinformatics 2014 [http://bioinformatics.oxfordjournals.org/content/early/2014/10/22/bioinformatics.btu704.short]

Howie B, Fuchsberger C, Stephens M, Marchini J, and Abecasis GR.
Fast and accurate genotype imputation in genome-wide association studies
through pre-phasing. Nature Genetics 2012 [http://www.nature.com/ng/journal/vaop/ncurrent/full/ng.2354.html]

= Questions and Comments =

Please contact [mailto:goncalo@umich.edu Goncalo Abecasis] or [mailto:cfuchsb@umich.edu Christian Fuchsberger].

= Related Pages =

If you are looking to learn about small computers made by Apple, Inc., you have come to the wrong page. Try looking at http://www.apple.com/macmini/, instead.

If you are looking for a low calorie version of the Big Mac sandwich, you'll be sad to know the Mini Mac has been discontinued. However, you are not the only one who likes the idea of a Mini Mac and you'll probably find some company on the web [http://mcchronicles.blogspot.com/2009/08/mini-mac-experiment-searching-for-big.html].

[[Category:Software]]

Minimac

2017-03-01T19:29:09Z

Pjvh:

'''This page is about the original version of Minimac. You may want to learn about new and improved [[Minimac3]].'''

'''minimac''' is a low memory, computationally efficient implementation of the MaCH algorithm for genotype imputation. It is designed to work on phased genotypes and can handle very large reference panels with hundreds or thousands of haplotypes. The name has two parts. The first, "mini", refers to the modest amount of computational resources it requires. The second, "mac", is short hand for [[MaCH]], our widely used algorithm for genotype imputation.

There are several minimac related pages on this wiki. The major ones are:

* [[Minimac]] - This page, the main minimac page.
* [[Minimac: Tutorial]] - A short minimac tutorial.
* [[Minimac: 1000 Genomes Imputation Cookbook]] - Cookbook instructions for 1000 Genomes Imputation with Minimac
* [[Minimac Command Reference]] - Summary of minimac options
* [[Minimac Diagnostics]] - Summary of diagnostics for imputation performance generated by minimac

*[https://imputationserver.sph.umich.edu Imputation server] - We are running imputation (and pre-phasing) for you!

= Download =
A binary Linux (64 bit) version of minimac is available [http://csg.sph.umich.edu/cfuchsb/minimac-beta-2013.7.17.tgz from here] and source code [http://csg.sph.umich.edu/cfuchsb/minimac.src.tgz from here]

The current version of minimac should be stamped 2013.7.17 - if your version shows a different version number or date stamp when it runs, it is not current.

If you use this beta version, please be sure to stop by the [http://www.sph.umich.edu/csg/abecasis/MaCH/download/ MaCH download page] and fill out the registration form, so that we can let you know when an official release is available and keep you updated with respect to any bug fixes.

== Multiprocessor Version ==

The current version of minimac comes in two flavours, <code>minimac</code> and <code>minimac-omp</code>. The latter version uses the [[OpenMP]] protocol to support multi-threading, resulting in faster throughput.

BE AWARE: since this version of minimac runs in parallel the order of samples in the output files (*dose, *haps,...) will vary between runs. Therefore, e.g. chunks have to be merged by sample id.

== Change log ==

2013.7.17

- minor bug fixes

-- all variants (SNPs, InDels, SVs) in the reference VCF will be imputed - independent from the FILTER column setting

- improved performance (Thanks to David Hinds - see also [[minimac2]] for the full set of performance improvements)

2012.11.16

- minor bug fixes

2012.10.9

- added: improved support for [http://www.shapeit.fr ShapeIT] phased haplotypes

2012.10.3

- added: full support for reference panel based chunking

2012.9.22

- fixed: chunk chromosome bug

2012.8.6 (early adopter)

- added: chromosome X support

2012.3.14

- fixed: problem with --startposition

2012.2.29

- added: VCF support

- added: IDR (Insertion, Deletion, Reference) support

== Questions and Comments ==

Please contact [mailto:goncalo@umich.edu Goncalo Abecasis] or [mailto:cfuchsb@umich.edu Christian Fuchsberger].

= Performance =

== Pre-phasing ==
For the pre-phasing step, cost increases quadratically with the number of states and linearly with the number of rounds. The following table provides a simple example.

{| class="wikitable" border="1" cellpadding="2"
|- bgcolor="lightgray"
! States
! Cost per round
|-
| 100 states
| 3 min
|-
| 200 states
| 12 min = (3 min * 22)
|-
| 400 states
| 48 min = (3 min * 42)
|-
| 500 states
| 75 min = (3 min * 52)
|}

So, in this case running haplotyping with 500 states and 10 rounds would require 75 min * 10 = 750 min.

Typically, haplotype quality improves rapidly with the number of states but only slowly with the number of rounds. We recommend running ~20 rounds of the MaCH haplotyper and selecting a number of states as high as your patience will allow (but ideally greater than 200).

== Imputation ==

A good rule of thumb is that minimac should take about 1 hour to impute 1,000,000 markers in 1,000 individuals using a reference panel with 100 haplotypes. Performance should scale linearly with respect to all these factors. So, your approximate computing time in hours should be about:

:<math>
E(\mbox{Run Time in Hours}) = N_{markers} * N_{individuals} * N_{haplotypes} * 10^{-11}
</math>

These statistics refer to a single core in a modern Intel CPU core and, although your mileage will vary, most modern CPUs should be no more than a few times faster (or slower) than that.

If you are estimating model parameters at the same time as imputing missing genotypes, you can account for the time needed for parameter estimation with the following formula:

:<math>
E(\mbox{Run Time in Hours}) = N_{markers} * ({N_{individuals} + N_{rounds} * N_{states} * 0.75 }) * N_{haplotypes} * 10^{-11}
</math>

In this updated formula, Nrounds represents the number of iterations used for parameter refinement and Nstates represents the maximum number of reference and target haplotypes considered for each update.

= Getting Started =

Using minimac for genotype imputation involves two steps. First, you will have to estimate haplotypes for your entire sample -- this will be the more computationally demanding step. Once that is done, you will be ready to quickly impute missing genotypes using the reference panel of your choice.

== Estimating Haplotypes for Your Sample ==

For the haplotyping step, we current recommend using [[MaCH]] with the --phase command line option. As input [[MaCH]] will need [[Merlin]] format pedigree and data files. All markers should be ordered according to their physical position and alleles should be labeled on the forward strand.

=== Preparing Your Data ===

To get started, you will need to store your data in [[Merlin]] format pedigree and data files, one per chromosome. For details, of the Merlin file format, see the [http://csg.sph.umich.edu/abecasis/Merlin/tour/input_files.html Merlin Tutorial].

Within each file, markers should be stored by chromosome position. Alleles should be stored in the forward strand and can be encoded as 'A', 'C', 'G' or 'T' (there is no need to use numeric identifiers for each allele).

We recommend that, if at all possible, you should phase your chromosomes according to NCBI build 37. Future releases of the 1000 Genomes Reference panel and other public sets of reference haplotypes are expected to use this genome build.

If figuring out position and strand for each marker seems like hard work, don't despair. For you, this should be the hardest bit of the entire process! For the computer, the fun is about to start.

==== NCBI build 36 / NCBI build 37 ====
The 1000G June reference panel is on build 36, future 1000G reference panels will be on build 37. This has also some impact on the pre-phasing:
some SNPs will ordered differently in each genome build (we expect order will be more accurate in more recent builds!). Purists will claim that imputation using NCBI build 36 and NCBI build 37 reference panels requires phasing using the exact same reference panel -- others will claim this makes little difference.

=== Running MaCH ===

A typical MaCH command line to estimate phased haplotypes might look like this:

mach1 -d sample.dat -p sample.ped --rounds 20 --states 200 --phase --interim 5 --sample 5 --compact

This will request that MaCH estimate haplotypes for your sample, using 20 iterations of its Markov sampler and conditioning each update on up to 200 haplotypes. A summary description of these parameters follows (but for a more complete description, you should go to the MaCH website):

{| class="wikitable" border="1" cellpadding="2"
|- bgcolor="lightgray"
! Parameter
! Description
|-
|style=white-space:nowrap|<code>-d sample.dat</code>
| Data file in [http://csg.sph.umich.edu/abecasis/Merlin/tour/input_files.html Merlin format]. Markers should be listed according to their order along the chromosome.
|-
| <code>-p sample.ped</code>
| Pedigree file in [http://csg.sph.umich.edu/abecasis/Merlin/tour/input_files.html Merlin format]. Alleles should be labeled on the forward strand.
|-
| <code>--states 200</code>
| Number of haplotypes to consider during each update. Increasing this value will typically lead to better haplotypes, but can dramatically increase computing time and memory use. A value of 200 - 400 is typical.
|-
| <code>--rounds 20</code>
| Iterations of the Markov sampler to use for haplotyping. Typically, using 20 - 30 rounds should give good results. To obtain better results, it is usually better to increase the <code>--states</code> parameter.
|-
| <code>--interim 5</code>
| Request that intermediate results should be saved to disk periodically. These will facilitate analyses in case a run doesn't complete.
|-
| <code>--sample 5</code>
| Request that random (but plausible) sets of haplotypes for each individual should be drawn every 5 iterations. This parameter is optional, but for some rare variant analyses, these alternative haplotypes can be very useful.
|-
| <code>--phase</code>
| Tell [[MaCH]] to estimate phased haplotypes for each individual.
|-
| <code>--compact</code>
| Reduce memory use at the cost of approximately doubling runtime. This option is recommended for most GWAS scale datasets and computing platforms.
|}

== Imputation into Phased Haplotypes ==

Imputing genotypes using '''minimac''' is an easy and straightforward process: after selecting a set of reference haplotypes, plugging-in the target haplotypes from the previous step and setting the number of rounds to use for the model parameter estimation, imputation should proceed rapidly.

=== Creating SNP List File ===

'''Minimac''' requires a file listing markers in the haplotype file. This file can be easily generated by extracting the second column from the .dat file. In a standard Unix system, a command like this should do:

cut -f 2 -d " " sample.dat > target.snps

=== Running Minimac ===

A typical minimac command line might look like this:

==== using a VCF reference panel ====
minimac --vcfReference --refHaps ref.vcf.gz --haps target.hap.gz --snps target.snps.gz --rounds 5 --states 200 --prefix results

'''Note''': GWAS SNPs (file --snps target.snps.gz) are by default expected to be in the chr:pos format e.g. 1:1000 and on build37/hg19; otherwise, please set the --rs flag

==== using a MaCH reference panel ====

minimac --refHaps ref.hap.gz --refSnps ref.snps.gz --haps target.hap.gz --snps target.snps.gz --rounds 5 --states 200 --prefix results

A detailed description of all minimac options is available [[Minimac Command Reference|elsewhere]]. Here is a brief description of the above parameters:

{| class="wikitable" border="1" cellpadding="2"
|- bgcolor="lightgray"
! Parameter
! Description
|-
| <code>--refSnps ref.snps.gz </code>
| List of SNPs in the reference panel
|-
| <code>--refHaps ref.hap.gz </code>
| Reference haplotypes (e.g. from [http://csg.sph.umich.edu/abecasis/MACH/download/ MaCH download page])
|-
| <code>--vcfReference </code>
| This option specifies that the provided --refHaps file is provided in VCF format , no --refSNPs file needed.
|-
| <code>--rs </code>
| In combination with --vcfReference, allows to use rs GWAS SNP identifiers
|-
| <code>--snps target.snps.gz </code>
| SNPs in phased haplotypes. These should largely be a subset of the SNPs in the reference panel.
|-
| <code>--haps target.hap.gz </code>
| Phased haplotypes where missing genotypes will be imputed.
|-
| <code>--sample target.sample </code>
| Sample list in ShapeIT format.
|-
| <code>--shape_haps target.hap.gz </code>
| ShapeIT phased haplotypes where missing genotypes will be imputed.
|-
| <code>--chr 22</code>
| Chromosome for which we will carry out imputation (needed to run ShapeIT with chr:pos identifiers - default setting).
|-
| <code>--rounds 5</code>
| Rounds of optimization for model parameters, which describe population recombination rates and per SNP error rates.
|-
| <code>--states 200</code>
| Maximum number of reference (or target) haplotypes to be examined during parameter optimization.
|-
| <code>--prefix imputed</code>
| Optionally, a string that is used to help generate output file names.
|}

=== Reference Haplotypes ===

Reference haplotypes generated by the 1000 Genomes project and formatted so that they are ready for analysis are available from the [http://csg.sph.umich.edu/abecasis/MACH/download/ MaCH download page]. As of this writing, the most recent set of haplotypes are based on genotype calls were generated in May 2011 and are an interim analysis of Project's Phase I data.

=== Imputation quality evaluation ===
To evaluate imputation quality, Minimac hides data for each genotyped SNP in turn and calculates 3 statistics:
* looRSQ - this is the estimated rsq for that SNP (as if SNP weren't typed).
* empR - this is the empirical correlation between true and imputed genotypes for the SNP. If this is negative, the SNP is probably flipped.
* empRSQ - this is the actual R2 value, comparing imputed and true genotypes.

These statistics can be found in the .info file

=== Additional Sources of Information ===

If the combination of MaCH and Minimac still runs too slowly for you, and you have access to a multi-processor compute cluster, you can look at [[ChunkChromosome]] page to learn how to conveniently split each chromosome into multiple segments that can be analyzed in parallel. For information on how to put the resulting chunks back together, see [[Ligate Minimac|this page]].

If you are especially interested in 1000 Genomes Imputation, then you should look at the [[Minimac: 1000 Genomes Imputation Cookbook]].

== X Chromosome Imputation ==
minimac supports the imputation of genotypes on the X chromosome (non-pseudo-autosomal part).

# Split the X chromosome pedigree file by sex.
## For females: follow the same protocol as for autosomes (phase with MaCH and impute with minimac).
## For males
### Remove the pseudo-autosomal part (for build hg18: chrX:1-2709520 and chrX:154584238-154913754 ; for build hg19 chrX:60001-2699520 and chrX:154931044-155260560)
### Convert the pedigree file into a MaCH haplotype file (missing genotypes should be encoded as: "0" or "." or "N" ) and impute using minimac as described above.

:::: '''<Example of a male only pedigree file>'''
:::: FAM1003 ID1234 0 0 M A/A A/A C/C
:::: FAM1004 ID5678 0 0 M 0/0 C/0 G/G
:::: ...
:::: '''<End of pedigree file>'''

:::: ''Note that, consistent with the Merlin convention, hemizygous males are listed as if they were homozygous.''

:::: '''<Example of the corresponding haplotype file>'''
:::: FAM1003->ID1234 HAPLO1 AAC
:::: FAM1003->ID1234 HAPLO2 AAC
:::: FAM1004->ID5678 HAPLO1 0CG
:::: FAM1004->ID5678 HAPLO2 0CG
:::: ...
:::: '''<End of the corresponding haplotype file>'''

= Post-imputation Association Analysis =
== Quantitative Traits ==
Please use [http://csg.sph.umich.edu/yli/mach/download/mach2qtl.source.V108.tgz mach2qtl].

== Binary Traits ==
Please use [http://csg.sph.umich.edu/yli/mach/download/mach2dat.source.1.0.18.tgz mach2dat]. Versions 1.0.18 and above accommodate to minimac output.

= Reference =

If you use [[minimac]] or [[minimac2]] please cite:

Fuchsberger C, Abecasis GR, Hinds DA. minimac2: faster genotype imputation. Bioinformatics 2014 [http://bioinformatics.oxfordjournals.org/content/early/2014/10/22/bioinformatics.btu704.short]

Howie B, Fuchsberger C, Stephens M, Marchini J, and Abecasis GR.
Fast and accurate genotype imputation in genome-wide association studies
through pre-phasing. Nature Genetics 2012 [http://www.nature.com/ng/journal/vaop/ncurrent/full/ng.2354.html]

= Questions and Comments =

Please contact [mailto:goncalo@umich.edu Goncalo Abecasis] or [mailto:cfuchsb@umich.edu Christian Fuchsberger].

= Related Pages =

If you are looking to learn about small computers made by Apple, Inc., you have come to the wrong page. Try looking at http://www.apple.com/macmini/, instead.

If you are looking for a low calorie version of the Big Mac sandwich, you'll be sad to know the Mini Mac has been discontinued. However, you are not the only one who likes the idea of a Mini Mac and you'll probably find some company on the web [http://mcchronicles.blogspot.com/2009/08/mini-mac-experiment-searching-for-big.html].

[[Category:Software]]

Minimac

2017-03-01T19:28:54Z

2015-10-27T22:08:15Z

Pjvh: /* Overview of the clipOverlap function of bamUtil */ Fix broken link.

[[Category:BamUtil|clipOverlap]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>clipOverlap</code> function of <code>bamUtil</code> =
The <code>clipOverlap</code> option on the [[bamUtil]] executable clips overlapping read pairs.

The input file and resulting output file is sorted by coordinate (or readName if specified in the options).

When a read is clipped from the front:
* the read start position is updated to reflect the clipping
* the mate's mate start position is updated to reflect the record's new position.
* the record is placed in the output file in the correct location based on the updated position.

To handle coordinate sorted files, SAM/BAM records are buffered up until it is known that all following records will have a later start position. To prevent the program from running away with memory, a limit is set to the number of records that can be buffered, see [[#Set the SAM/BAMs record buffer size (--poolSize)|<code>--poolSize</code>]] for more information.

When two mates overlap, this tool will clip the record's whose clipped region would has the lowest average quality.

It also checks strand. If a forward strand extends past the end of a reverse strand, that will be clipped. Similarly, if a reverse strand starts before the forward strand, the region prior to the forward strand will be clipped. If the reverse strand occurs entirely before the forward strand, both strands will be entirely clipped. If the [[#Mark entirely clipped reads as unmapped (--unmapped)|<code>--unmapped</code>]] option is specified rather than clipping an entire read, it will be marked as unmapped.

The qualities on the two strands remain unchanged even with clipping.

== ASSUMPTIONS/RESTRICTIONS ==

*Assumes the file is sorted by Coordinate (or ReadName if using <code>--readName</code> option)
*Assumes only 2 reads have matching ReadNames
**It matches in pairs, so if there are 3, the first 2 will be matched and compared, but the 3rd won't. If there are 4, the first 2 will be matched and the last 2 will be matched and compared.
*Only mapped reads will be clipped
*Mate information in records are accurate

= Rules for Clipping =

== Clipping from the front==
The first operation after the softclip will be a Match/Mismatch, meaning that any trailing pads, deletions, insertions, or skips will also be soft clipped.
{|border="1" cellspacing="0" cellpadding="2"
! Clip Location !! How it is handled
|-
|If the clip position falls in a skip/deletion
| Removes the entire skip/deletion
|-
|If the position immediately after the clip is a skip/deletion
| Also removes the skip/deletion
|-
|If the position immediately after the clip is an Insert
| Softclips the insert
|-
|If the position immediately after the clip is a Pad
| Removes the pad
|-
| Clip occurs at the last match/mismatch position of the read (the entire read is clipped)
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs after the read ends
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs before the read starts
| Nothing is clipped. The read is not changed.
|}

== Clipping from the back==

{|border="1" cellspacing="0" cellpadding="2"
! Clip Location !! How it is handled
|-
|If the clip position falls in a skip/deletion
| Removes the entire skip/deletion
|-
|If the position immediately before the clip is a deletion/skip/pad
| Remove the deletion/skip/pad
|-
|If the position immediately before the clip is an insertion
| Leave the insertion, even if it results in a 70M3I27S
|-
|Clip occurs at the first position of the read (the entire read is clipped)
| Entire read is soft clipped, preceding insertions remain, 0-based position is left as the original (not modified)
|-
| Clip occurs before the read starts
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs after the read ends
| Nothing is clipped. The read is not changed.
|}

= Usage =
./bam clipOverlap --in <inputFile> --out <outputFile> [--storeOrig <tag>] [--readName] [--stats] [--overlapsOnly] [--excludeFlags <flag>] [--poolSize <numRecords allowed to allocate>] [--poolSkipOverlap] [--noeof] [--params]

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to clip overlaping read pairs for
--out : the SAM/BAM file to be written
Optional Parameters:
--storeOrig : Store the original cigar in the specified tag.
--readName : Original file is sorted by Read Name instead of coordinate.
--stats : Print some statistics on the overlaps.
--overlapsOnly : Only output overlapping read pairs
--excludeFlags : Skip records with any of the specified flags set, default 0x70C
--unmapped : Mark records that would be completely clipped as unmapped
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings to stderr
Clipping By Coordinate Optional Parameters:
--poolSize : Maximum number of records the program is allowed to allocate
for clipping on Coordinate sorted files. (Default: 1000000)
--poolSkipClip : Skip clipping reads to free of usable records when the
poolSize is hit. The default action is to just clip the
first read in a pair to free up the record.
</pre>
{{PhoneHomeParamDesc}}

== Required Parameters==
{{inBAMInputFile}}
{{outBAMOutputFile}}

== Optional Parameters ==
=== Store the original cigar string in a tag (<code>--storeOrig</code>) ===

Use <code>--storeOrig</code> followed by the two character TAG to store the original CIGAR.

It will be stored with the specified tag as a "Z" tag type.

=== Work on SAM/BAMs sorted by Read Name instead of by coordinate (<code>--readName</code>) ===

If your file is sorted by read name rather than by coordinate, specify <code>--readName</code>. The resulting file will still be sorted by read name.

=== Print Overlap Statistics (<code>--stats</code>)===
Print some basic overlap statistics to stderr.

Output values
* count of the number of overlapping pairs that are clipped
* average of the number of overlapping reference bases that are clipped
* variance of the number of overlapping reference bases that are clipped
* number of times the forward strand is clipped when read pairs overlap
* number of times the reverse strand is clipped when read pairs overlap
* number of times the orientation causes clipping/additional clipping
** reads that are only clipped due to orientation are not counted in the other stats

==== Example Output ====
<pre>
Overlap Statistics:
Number of overlapping pairs: 14
Average # Reference Bases Overlapped: 18.3571
Variance of Reference Bases overlapped: 39.1703
Number of times the forward strand was clipped: 6
Number of times the reverse strand was clipped: 8
Number of times orientation causes additional clipping: 4
</pre>

=== Print Only Overlaping Reads (<code>--overlapsOnly</code>)===
Only output Read Pairs that overlap. Drop all other records.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0x70C

By default skips reads with any of the following flags set:
* unmapped
* mate unmapped
* secondary alignment
* fails QC checks
* duplicate

This parameter was added in version 1.0.10.

=== Mark entirely clipped reads as unmapped (<code>--unmapped</code>)===
Specify this option if instead of marking reads as entirely clipped, mark them as unmapped.

When marking a read as unmapped, it will:
* Set CIGAR to 0
* Set MapQ to 0
* Clear N/A flag fields:
** Proper pair
** Secondary Alignment
** Supplementary Alignment
* Update the Mate's flag to indicate:
** Mate Unmapped
** Not proper pair

{{noeofBGZFParameter}}
{{paramsParameter}}

==Clipping By Coordinate Optional Parameters==
=== Set the SAM/BAMs record buffer size (<code>--poolSize</code>) ===

To handle coordinate sorted files, SAM/BAM records are buffered until it is known that all following records will have a later start position. To prevent the program from running away with memory, a limit is set to the number of records that can be buffered (defaults to 1000000).

If the poolSize is exhausted, the code will write the earliest record awaiting its overlapping mate and any previous records that are being buffered.

Depending on whether or not <code>--poolSkipClip</code> is set, it will either, clip the end of the read at the position where the mate is supposed to start or it will not clip either read. An error message is written to stderr to indicate that one of these has happened and an unsuccessful return value is returned (2: NO_MORE_RECS).

The resulting file will still be sorted by coordinate.

=== Skip Clipping Coordinate Sorted Files When Out of Records (<code>--poolSkipClip</code>) ===

When clipping coordinate sorted SAM/BAM files, we can run out of buffers available in the pool (<code>--poolSize</code>).

By default when we run out of pooled records, we can no longer read in new records, so instead we release some of the stored records. We do this by dropping the first record that is being held awaiting its mate.

This record can either be:
* Clipped starting at its mate's start position until the end of the read (DEFAULT)
* Left as is with no clipping, leaving the mates mates overlapping (specify <code>--poolSkipClip</code>)

With either option, the resulting file will still be sorted by coordinate.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 for success, non-0 for failure).

Returns SamStatus::NO_MORE_RECS, 2, if it was clipping files sorted by coordinate and it ran out of records in the pool so had to clip based on the <code>--poolSkipClip</code> setting.

= Output =

All status messages are written to stderr.

{|border="1" cellspacing="0" cellpadding="2"
! Situation !! Sorted Type !! Output Message
|-
| Everything ran successfully
| ReadName/Coordinate
|
Completed ClipOverlap Successfully.
|-
| Failed to allocate any records
| ReadName/Coordinate
|
Failed to allocate any records.
Failed to complete ClipOverlap.
|-
| Error writing a record
| ReadName
|
Failed to complete ClipOverlap.
|-
| Expected pair to overlap, but 2nd read was not found in the specified position (may be combined with output for running out of pooled records)
| Coordinate
|
Failed to find expected overlapping mates for XX records.
Completed ClipOverlap Successfully.
|-
| Ran out of pooled Records with <code>--poolSkipClip</code> setting
| Coordinate
|
Due to hitting the max record poolSize, had to skip clipping XX records.
Completed ClipOverlap.
|-
| Ran out of pooled Records without <code>--poolSkipClip</code> setting
| Coordinate
|
Due to hitting the max record poolSize, had to default clip XX records.
Completed ClipOverlap.
|-
| Any other error
| Coordinate
|
Failed to complete ClipOverlap.
|}

== Example Output ==
<pre>
Failed to find expected overlapping mates for 2 records.
Due to hitting the max record poolSize, had to default clip 9 records.
Completed ClipOverlap.
</pre>

2015-08-31T17:55:52Z

Pjvh: matching parenthesis

Back to Parent: [[GotCloud]]

==Instructions for running on Amazon==

'''[[Amazon Single Node]]''' : Instructions for setting up a single node with the GotCloud AMI.
* Run the [[GotCloud: Amazon Demo]] (includes instructions for setting up a single node.)

'''[[Amazon Storage]]''' : Instructions for setting up storage
* Not all EC2 instances contain a lot of storage, so you may need more
* Clusters may need common storage

'''[[StarCluster]]''' : Instructions for setting up a cluster

'''Miscellaneous Instructions''':
* If you don't want to use the GotCloud AMI, you can install GotCloud as you would on any system, following the steps at: [[GotCloud:_Source_Releases#Downloading_From_GitHub_Without_Git|Downloading GotCloud]]

'''Developer Instructions''':
* [[Creating an AMI on EC2]]
* Snapshot (deprecated)
** [[Amazon Snapshot]]
** [[Creating a Snapshot on EC2]]
* Old UM setup info: [[AWS Resources]]