Genome Analysis Wiki - User contributions [en]

Minimac4

2023-06-14T15:40:41Z

Lefaivej:

= IMPORTANT NOTE !!! =

This wiki is outdated and only applicable to version 4.0.x (a.k.a 1.0.x). Documentation for version 4.1.x can be found on the [https://github.com/Santy-8128/Minimac4 Minimac4 Github page].

= Introduction =

'''Minimac4 ''' is a latest version in the series of genotype imputation software - preceded by [[Minimac3|Minimac3]] (2015), [[Minimac2|Minimac2]] (2014), [[Minimac|minimac]] (2012) and [[MaCH|MaCH]] (2010). '''Minimac4''' is a lower memory and more computationally efficient implementation of the original algorithms with comparable imputation quality.

The Minimac3 mailing list has been renamed as the Minimac4 mailing list. If you were already a member, no need to re-join. If not, please join our [https://groups.google.com/forum/embed/?place=forum/minimac4-help&umich.edu| mailing list] to get updates about future releases or report possible bugs or email them to [mailto:yukt@umich.edu Ketian Yu] or [mailto:sayantan@umich.edu Sayantan Das].'''

= Installation =

'''Minimac4 (version 1.0.0, released 2.14.2018)''' is currently available on [https://github.com/Santy-8128/Minimac4 Minimac4 Github].

The easiest way to install Minimac4 and its dependencies is to use the install.sh file provided.
git clone <nowiki>https://github.com/statgen/Minimac4.git</nowiki>
cd Minimac4
bash install.sh

Please see [https://github.com/Santy-8128/Minimac4 Minimac4 Github] for the full instructions for installation.

Commonly used reference panels in M3VCF format are available for download in [[#Reference Panels for Download | Reference Panels]].

= What's New =

The input file format, output file formats and typical command lines are the same in Minimac4 (as they were in minimac3). Some of the main new features are summarized below:

* '''Improved Speed - ''' Minimac4 is approximately '''6 times''' faster for 1000 Genomes Phase 1 and Phase 3 and '''2 times''' faster for the HRC reference panels at comparable accuracy (details of accuracy for imputing into 10 European samples are given here). The speed can be further improved by tuning the approximation parameters (see below), but we recommend using the default values.

* '''Automated Chunking - ''' Minimac4 automatically chunks the whole chromosome (into overlapping chunks), analyzes each chunk sequentially and then concatenates the imputed chunks back. This caps the memory usage across different chromosomes (memory requirement is based on chunk size, not chromosome size). The length of the chunk and the overlap can be controlled by the <code>--chunkLengthMb</code> and <code>--chunkLengthOverlapMb</code> options, although we recommend using the default values of 20 and 3, respectively.

* '''Approximations - ''' Minimac4 uses some simple approximations to speed up the imputation analyses. The levels of approximation can be controlled by the parameters <code>--probThreshold</code>, <code>--diffThreshold</code>, and <code>--topThreshold</code> (details given in Minimac4 Usage). Higher levels of approximation will reduce the compute time but marginally reduce the imputation accuracy. We recommend using the default values (0.01).

* '''Improved Chromosome X/Y Support - ''' Minimac4 can handle different ploidys in the same VCF file for imputation of sex chromosomes. For example, for the non-PAR region on chromosome X, males and females can be imputed together, irrespective of whether males are coded as haploids or diploids. However, each sample must have a fixed ploidy. Thus, PAR and non-PAR regions still need to be imputed separately. Please see Chromosome X Imputation for more details.

* '''Other Helpful Features'''
** We introduced a new feature called <code>--memUsage</code> that will estimate and report the memory required by Minimac4. This feature should be useful for users running their jobs on a compute cluster that requires memory specification.
** We introduced some other FORMAT options for the output dosage data, allowing users to retrieve haplotype dosages, genotype probabilities, genotype dosages or any other measure of summary that they want.
** We have fixed the bug related to FILTER=GENOTYPED and FILTER=GENOTYPED_ONLY which was causing a crash in bcftools.

* '''Obsolete Features'''
** In Minimac4, we removed the <code>--doseOutput</code> and <code>--hapOutput</code> options. Please use [[DosageConvertor]] to convert your files to MaCH or PLINK dosage format.
** Currently Minimac4 can ONLY handle M3VCF format files. If your reference panel is in VCF format, please use [[Minimac3]] to convert the VCF file to M3VCF (along with parameter estimation) and then use that M3VCF for imputation using Minimac4. The same holds for the option <code>--processReference</code> as well. Although the handle is made available, we will implement it in a later version.
** Parameters such as <code>--rounds</code>, <code>--states</code>, <code>--rec</code>, <code>--err</code> have been deactivated for now until we implement parameter estimation in minimac4.

= Usage =
A typical Minimac4 command line for imputation is as follows

minimac4 --refHaps refPanel.m3vcf \
--haps targetStudy.vcf \
--prefix testRun

Here refPanel.m3vcf is the reference panel used in M3VCF format (e.g. 1000 Genomes), targetStudy.vcf is the phased GWAS data in VCF format, and testRun is the prefix for the output files.

=== Full List of Options ===
Please see '''[[Minimac4 Documentation]]''' for detailed explanation of all available options.

Also, users can always type the following for the full list of available options:
minimac4 --help

=== Convert VCF to M3VCF ===
If the reference panel is in VCF format, please use [[Minimac3]] to convert it into M3VCF format first.
../bin/Minimac3 --refHaps refPanel.vcf \
--processReference \
--prefix refPanel

=== Multi-Threading ===
The following example shows the same analysis as above, but using 5 threads:

minimac4 --refHaps refPanel.m3vcf \
--haps targetStudy.vcf \
--prefix testRun \
--cpus 5

= Reference Panels for Download =

Some commonly used reference panels are available for download here:

{| class="wikitable" style="text-align:center" border="1" cellpadding="2"
|- bgcolor="lightgray"
! width="150px" |Reference Panel
! width="100px" |Number of Samples
! width="100px" |File Format
! width="100px" |Parameter Estimates Available
! width="120px" |Chromosomes
! width="80px" |Link
|-
| rowspan=4 | '''1000 Genomes''' 
'''Phase 3''' 
(version 5)
| rowspan=4 style="text-align:center" | '''2,504'''
| '''VCF'''
| -
| 1-22,X
| [ftp://share.sph.umich.edu/minimac3/G1K_P3_VCF_Files.tar.gz Download] 
|-
| rowspan=2 style="text-align:center" | '''M3VCF'''
| YES
| 1-22,X
| [ftp://share.sph.umich.edu/minimac3/G1K_P3_M3VCF_FILES_WITH_ESTIMATES.tar.gz Download] 
|-
|NO
| 1-22,X
| [ftp://share.sph.umich.edu/minimac3/G1K_P3_M3VCF_FILES_NO_ESTIMATES.tar.gz Download] 
|-
| '''VCF''','''M3VCF'''
| YES
| X
| [ftp://share.sph.umich.edu/minimac3/G1K_P3_CHR_X_VCF_M3VCF_FILES.tar.gz Download] 
|-
| rowspan=4 | '''1000 Genomes''' 
'''Phase 1''' 
(version 3)
| rowspan=4 | '''1,092'''
| '''VCF'''
| -
| 1-22,X
| [ftp://share.sph.umich.edu/minimac3/G1K_P1_VCF_Files.tar.gz Download]
|-
| rowspan=2 style="text-align:center" | '''M3VCF'''
| YES
| 1-22,X
| [ftp://share.sph.umich.edu/minimac3/G1K_P1_M3VCF_FILES_WITH_ESTIMATES.tar.gz Download]
|-
| NO
| 1-22,X
| [ftp://share.sph.umich.edu/minimac3/G1K_P1_M3VCF_FILES_NO_ESTIMATES.tar.gz Download]
|-
| '''VCF''','''M3VCF'''
| YES
| X
| [ftp://share.sph.umich.edu/minimac3/G1K_P1_CHR_X_VCF_M3VCF_FILES.tar.gz Download] 
|}

= Useful Wiki Pages =

There are a few pages in this Wiki that may be useful to for '''Minimac4''' users. Here are links to a few:

* [[Minimac4| Minimac4 Overview Page]]

* [[Minimac4 Documentation]]

* [[M3VCF Files| M3VCF Files]]

[[Category:Software]]

SAV File Format

2018-05-22T14:03:45Z

Lefaivej:

SAV (Sparse Allele Vectors) is a file format for storing very large sets of genotypes and halpotype dosages that produces small file sizes and is optimized for fast association analysis. The design supports fine-level random access to variants, with no limitations on variant annotation, sample size, ploidy level or genome length.

SAV capitalizes on the sparsity of genetic variation to both compress data and reduce deserialization overhead. Since the proportion of rare variants continually increases with sample size, the compression ratio and efficiency of our format both improve as study sizes grow. In addition to the I/O and computational efficiency attributed to reduced storage footprints, further computational efficiency can be achieved through sparse matrix operations.

Accompanying the SAV format is the [[Savvy C++ Library | Savvy C++ programming library ]] for interfacing with it and other file formats. This library was designed for efficient association analysis and provides a mechanism to plug in linear algebra and numerical libraries, which reduces the overhead of copying data and lowers the memory footprint.

==S1R Index==
SAV files are indexed using an S1R (Sort-tile-recursive One-dimensional R-tree) index file. Genomic regions are organized into an r-tree to enable fast random access to a SAV file without having to parse the entire index file. Each leaf entry in the tree points to a zstd compressed block in the corresponding SAV file. The entry also encodes the number of variants in the block, which can be variable depending on the parameters for compressing the SAV file.

===Diagram of an S1R R-tree===
[[File:s1r_diagram.png|900 px]]

== APIs ==
* [[Savvy C++ Library | Savvy]] (C++)
* [[SAV Command Line Interface | sav]] (CLI)
* [[SavR]] (R)

SAV File Format

2018-05-22T14:01:43Z

Lefaivej:

SAV (Sparse Allele Vectors) is a file format for storing very large sets of genotypes and halpotype dosages that produces small file sizes and is optimized for fast association analysis. The design supports fine-level random access to variants, with no limitations on variant annotation, sample size, ploidy level or genome length.

SAV capitalizes on the sparsity of genetic variation to both compress data and reduce deserialization overhead. Since the proportion of rare variants continually increases with sample size, the compression ratio and efficiency of our format both improve as study sizes grow. In addition to the I/O and computational efficiency attributed to reduced storage footprints, further computational efficiency can be achieved through sparse matrix operations.

Accompanying the SAV format is the [[Savvy C++ Library | Savvy C++ programming library ]] for interfacing with it and other file formats. This library was designed for efficient association analysis and provides a mechanism to plug in linear algebra and numerical libraries, which reduces the overhead of copying data and lowers the memory footprint.

==S1R Index==
SAV files are indexed using an S1R (Sort-tile-recursive One-dimensional R-tree) index file. Genomic regions are organized into an r-tree to enable fast random access to a SAV file without having to parse the entire index file. Each leaf entry in the tree points to a zstd compressed block in the corresponding SAV file. The entry also encodes the number of variants in the block, which can be variable depending on the parameters for compressing the SAV file.

===Diagram of R-tree===
[[File:s1r_diagram.png|900 px]]

== APIs ==
* [[Savvy C++ Library | Savvy]] (C++)
* [[SAV Command Line Interface | sav]] (CLI)
* [[SavR]] (R)

File:S1r diagram.png

2018-05-22T13:49:48Z

Lefaivej: Visual representation of and s1r index tree.

Visual representation of and s1r index tree.

SAV Command LIne Interface

2018-05-22T13:34:20Z

Lefaivej: Lefaivej moved page SAV Command LIne Interface to SAV Command Line Interface: Typo

#REDIRECT [[SAV Command Line Interface]]

SAV Command Line Interface

2018-05-22T13:34:20Z

Lefaivej: Lefaivej moved page SAV Command LIne Interface to SAV Command Line Interface: Typo

The sav application is a command line Interface for manipulating [[SAV File Format | sav]] genotype files.

==Usage==
Running <code>sav --help</code> will list available subcommands. Each subcommand also has its own help page. When installed into systems paths, <code>man sav</code> will also bring up documentation.

===import===
The <code>import</code> subcommand imports from BCF or VCF to SAV file format. Indexing, sorting, subsetting, etc. can all be applied during import. GT is the default data format imported. HDS (haplotype dosages) can be enabled with <code>-d HDS</code>.

<syntaxhighlight lang="shell">
# imports BCF file and indexes output.
sav import --index file.bcf file.sav
</syntaxhighlight>

===export===
The <code>export</code> subcommand will export a SAV file to either plain text or bgzipped VCF format. The exported data format can be GT, HDS, GP or DS.
<syntaxhighlight lang="shell">
# exports region and sample subset to stdout
sav export file.sav \
--regions chr1,chr2:10000-20000 \
--sample-ids ID1,ID2,ID3 | less -S
</syntaxhighlight>

<syntaxhighlight lang="shell">
# exports sample subset as dosages excluding monomorphic sites to bgzf file.
sav export file.sav file.vcf.gz \
--filter "AC>0;AC!=AN" \
--data-format DS \
--sample-ids ID1,ID2,ID3
</syntaxhighlight>

===stat-index===
The <code>stat-index</code> subcommand can be used to quickly pull metrics about the call set (eg. number of variants) via the S1R index.

<syntaxhighlight lang="shell">
sav stat-index file.sav.s1r
> chromosome 18 20
> block count 4813 3421
> marker count 9855090 8835691
> tree height 2 2
</syntaxhighlight>

==External links==
[https://github.com/statgen/savvy| Github repo] (source code)

SAV Command Line Interface

2018-05-22T13:16:34Z

Lefaivej:

SAV Command Line Interface

2018-05-22T13:12:59Z

Lefaivej:

SAV File Format

2018-05-21T14:50:18Z

Lefaivej: /* APIs */

SAV Command Line Interface

2018-05-21T14:50:15Z

Lefaivej: Created page with "The sav application is a command line Interface for manipulating sav genotype files. [https://github.com/statgen/savvy| Github repo]"

The sav application is a command line Interface for manipulating [[SAV File Format | sav]] genotype files.

[https://github.com/statgen/savvy| Github repo]

SAV File Format

2018-05-21T14:46:31Z

Lefaivej: /* APIs */

SAV File Format

2018-04-20T16:52:53Z

Lefaivej: /* APIs */

SAV File Format

2018-04-19T20:19:14Z

Lefaivej:

SavR

2018-04-19T20:19:04Z

Lefaivej: Created page with "SavR is an R package for interfacing with SAV files. [https://github.com/statgen/savr| Github repo]"

SavR is an R package for interfacing with [[SAV File Format | SAV]] files.

[https://github.com/statgen/savr| Github repo]

SAV File Format

2018-04-19T20:14:56Z

Lefaivej:

SAV File Format

2018-04-19T19:51:39Z

Lefaivej:

Savvy

2018-04-19T19:50:37Z

Lefaivej: Lefaivej moved page Savvy to Savvy C++ Library

#REDIRECT [[Savvy C++ Library]]

Savvy C++ Library

2018-04-19T19:50:36Z

Lefaivej: Lefaivej moved page Savvy to Savvy C++ Library

The Savvy C++ library is an Interface to various variant calling formats, most notably [[SAV File Format]].

[https://github.com/statgen/savvy| Github repo]

Savvy C++ Library

2018-04-19T19:49:47Z

Lefaivej:

The Savvy C++ library is an Interface to various variant calling formats, most notably [[SAV File Format]].

[https://github.com/statgen/savvy| Github repo]

SAV File Format

2018-04-19T19:48:17Z

Lefaivej: /* SAV (Sparse Allele Vectors) */

Savvy C++ Library

2018-04-19T19:46:41Z

Lefaivej: /* Savvy C++ Library */

= Savvy C++ Library =

Interface to various variant calling formats.

[https://github.com/statgen/savvy| Github repo]

Savvy C++ Library

2018-04-19T19:45:35Z

Lefaivej: Created page with "= Savvy C++ Library = Interface to various variant calling formats. [https://github.com/statgen/savvy]"

= Savvy C++ Library =

Interface to various variant calling formats.

[https://github.com/statgen/savvy]

SAV File Format

2018-04-19T19:42:46Z

Lefaivej:

= SAV (Sparse Allele Vectors) =

SAV is a file format for storing very large sets of genotypes and halpotype dosages that produces small file sizes and is optimized for fast association analysis. The design supports fine-level random access to variants, with no limitations on variant annotation, sample size, ploidy level or genome length.

SAV capitalizes on the sparsity of genetic variation to both compress data and reduce deserialization overhead. Since the proportion of rare variants continually increases with sample size, the compression ratio and efficiency of our format both improve as study sizes grow. In addition to the I/O and computational efficiency attributed to reduced storage footprints, further computational efficiency can be achieved through sparse matrix operations.

Accompanying the SAV format is the [[Savvy]] C++ programming library for interfacing with it and other file formats. This library was designed for efficient association analysis and provides a mechanism to plug in linear algebra and numerical libraries, which reduces the overhead of copying data and lowers the memory footprint.

SAV File Format

2018-04-19T19:38:00Z

Lefaivej: Created page with "= SAV (Sparse Allele Vectors) = SAV is a file format for storing very large sets of genotypes and halpotype dosages that produces small file sizes and is optimized for fast a..."

= SAV (Sparse Allele Vectors) =

SAV is a file format for storing very large sets of genotypes and halpotype dosages that produces small file sizes and is optimized for fast association analysis. The design supports fine-level random access to variants, with no limitations on variant annotation, sample size, ploidy level or genome length.

SAV capitalizes on the sparsity of genetic variation to both compress data and reduce deserialization overhead. Since the proportion of rare variants continually increases with sample size, the compression ratio and efficiency of our format both improve as study sizes grow. In addition to the I/O and computational efficiency attributed to reduced storage footprints, further computational efficiency can be achieved through sparse matrix operations.

Accompanying the SAV format is a C++ programming library (Savvy) for interfacing with it and other file formats. This library was designed for efficient association analysis and provides a mechanism to plug in linear algebra and numerical libraries, which reduces the overhead of copying data and lowers the memory footprint.