Genome Analysis Wiki - User contributions [en]

BamUtil: recab

2019-04-20T03:01:33Z

Mktrost: /* Required Generic Parameters */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>recab</code> function of <code>[[bamUtil]]</code> =
The <code>recab</code> option of [[bamUtil]] recalibrates a SAM/BAM file.

Recalibration can also be called as an option of [[bamUtil: dedup]]. This will perform the recalibration and the deduping in the same set of steps, increasing processing speed.

==Handling Recalibration/Implementation Notes==

Recalibration is a 2-step process that loops through the file twice (stdin is not support as input):
# Build Recalibration Table
# Apply Recalibration Table

The Recalibration Table groups bases based on a set of covariates:
* Read Group
* Quality (either from the quality string or [[#Read the quality from a tag (--qualField)|from a tag]])
* Cycle (reverse complement for reverse strands)
* 1st/2nd read in pair
* Previous Cycle's Base (reverse complement for reverse strands)
* This Cycle's Base (reverse complement for reverse strands)

The Recalibration Table tracks the number of matches/mismatches for each set of covariates.

Only bases meeting all of the following criteria are used to Build the Recalibration Table:
* Read criteria
** not a duplicate
** mapped
** mapping quality != 0
** mapping quality != 255
* Base criteria
** match/mismatch (not an insertion/deletion/skip/clip)
** not a [[#DBSNP File (--dbsnp)|dbSNP position]]
** base quality > [[#Minimum Recalibration Base Quality (--minBaseQual)|minBaseQual (5 by default)]]
* Additional criteria for cycle != 1 (can be turned off via flags)
** previous base is a CIGAR Match/Mismatch (Use [[#Allow Previous Base Non-Match/Mismatch (--keepPrevNonAdjacent)|<code>--keepPrevNonAdjacent</code>]] to disable)
** previous base position is not a [[#DBSNP File (--dbsnp)|dbSNP position]] (Use [[#Allow Previous Base DBSNP (--keepPrevDbsnp)|<code>--keepPrevDbsnp</code>]] to disable)

The Recalibration Table is applied to all bases meeting all of the following criteria (even if they were not used for creating the table):
* base quality > [[#Minimum Recalibration Base Quality (--minBaseQual)|minBaseQual (5 by default)]]
* at least 1 match or mismatch for the set of covariates

Recalibrated Quality is: <math>-10 * \log \frac{mismatches + 1}{mismatches + matches + 1}</math>

Alternatively, [[#Logistic Regression (--useLogReg)|logistic regression]] can be used for calculating the new quality.

If the Recalibrated Quality is greater than [[#Maximum Recalibration Base Quality (--maxBaseQual)|maxBaseQual]], the updated quality is set to maxBaseQual.

Optionally, the previous quality can be [[#Store the original quality (--storeQualTag)|stored in a tag]].

The current recalibration logic was designed for recalibrating ILLUMINA data.

NOTE: GATK ignores/skips adapters, but our logic does not.

== How to use it ==

When <code>recab</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file ([[#input File (--in)|--in]]), the output SAM/BAM file ([[#output File (--out)|--out]]), and the reference file ([[#Reference File (--refFile)|--refFile]]) are required inputs.

Recommended usage with Deduper:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Deduper:

/usr/cluster/bin/bam recab --in ${INPUT}.bam --out ${OUTPUT}.bam --refFile ${REF} --dbsnp ${DBSNP} --storeQualTag OQ --maxBaseQual 40

= Usage =
./bam recab (options) --in <InputBamFile> --out <OutputFile> [--log <logFile>] [--verbose] [--noeof] [--params] --refFile <ReferenceFile> [--dbsnp <dbsnpFile>] [--minBaseQual <minBaseQual>] [--maxBaseQual <maxBaseQual>] [--blended <weight>] [--fitModel] [--fast] [--keepPrevDbsnp] [--keepPrevNonAdjacent] [--useLogReg] [--qualField <tag>] [--storeQualTag <tag>] [--buildExcludeFlags <flag>] [--applyExcludeFlags <flag>]

= Parameters =
<pre>
Required General Parameters :
--in <infile> : input BAM file name
--out <outfile> : output recalibration file name
Optional General Parameters :
--log <logfile> : log and summary statistics (default: [outfile].log)
--verbose : Turn on verbose mode
--noeof : do not expect an EOF block on a bam file.
--params : print the parameter settings

Recab Specific Required Parameters
--refFile <reference file> : reference file name
Recab Specific Optional Parameters :
--dbsnp <known variance file> : dbsnp file of positions
--minBaseQual <minBaseQual> : minimum base quality of bases to recalibrate (default: 5)
--maxBaseQual <maxBaseQual> : maximum recalibrated base quality (default: 50)
qualities over this value will be set to this value.
This setting is applied after binning (if applicable).
--blended <weight> : blended model weight
--fitModel : check if the logistic regression model fits the data
overriden by fast, but automatically applied by useLogReg
--fast : use a compact representation that only allows:
* at most 256 Read Groups
* maximum quality 63
* at most 127 cycles
overrides fitModel, but is overridden by useLogReg
uses up to about 2.25G more memory than running without --fast.
--keepPrevDbsnp : do not exclude entries where the previous base is in dbsnp when
building the recalibration table
By default they are excluded from the table.
--keepPrevNonAdjacent : do not exclude entries where the previous base is not adjacent
(not a Cigar M/X/=) when building the recalibration table
By default they are excluded from the table (except the first cycle).
--useLogReg : use logistic regression calculated quality for the new quality
automatically applies fitModel and overrides fast.
--qualField <quality tag> : tag to get the starting base quality
(default is to get it from the Quality field)
--storeQualTag <quality tag> : tag to store the previous quality into
--buildExcludeFlags <flag> : exclude reads with any of these flags set when building the
recalibration table. Default is 0xF04
--applyExcludeFlags <flag> : do not apply the recalibration table to any reads with any of these flags set
Quality Binning Parameters (optional):
Bin qualities by phred score, into the ranges specified by binQualS or binQualF (both cannot be used)
Ranges are specified by comma separated minimum phred score for the bin, example: 1,17,20,30,40,50,70
The first bin always starts at 0, so does not need to be specified.
By default, the bin value is the low end of the range.
--binQualS : Bin the Qualities as specified (phred): minQualOfBin2, minQualofBin3...
--binQualF : Bin the Qualities based on the specified file
--binMid : Use the mid point of the quality bin range for the quality value of the bin.
--binHigh : Use the high end of the quality bin range for the quality value of the bin.

</pre>
{{PhoneHomeParamDesc}}

== Required Generic Parameters ==
{{inBAMInputFile|noStdin=1}}
{{outBAMOutputFile}}

== Optional Generic Parameters ==
=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

{{PhoneHomeParameters}}

== Required Recalibration Parameters ==
=== Reference File (<code>--refFile</code>) ===

The reference file is a required parameter used for comparing read bases to the reference.

== Optional Recalibration Parameters ==

=== DBSNP File (<code>--dbsnp</code>) ===

The dbsnp file that specifies positions to skip recalibrating. Tab delimited file with the chromosome in the first column and the 1-based position in the 2nd column.

=== Minimum Recalibration Base Quality (<code>--minBaseQual</code>) ===

When recalibrating reads, only positions with a base quality greater than this minimum phred quality will be recalibrated. If <code>--minBaseQual</code> is not specified, it is defaulted to 5.

The ILLUMINA specs indicate that any quality below 5 can be used as an error indicator so we do not want to recalibrate those.

=== Maximum Recalibration Base Quality (<code>--maxBaseQual</code>) ===

This value sets the maximum phred base quality assigned to a base after recalibrating. Any qualities above this value will be set to this value. It is defaulted to 50.

=== Blended Model Weight (<code>--blended</code>) ===

TBD - this parameter is not yet implemented.

=== Fit Model (<code>--fitModel</code>) ===

Check if the logistic regression model fits the data.

This option does NOT set the new qualities to the logistic regression calculated qualities, it only checks the fit. To apply the logistic regression qualities, see [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]]. <code>--fitModel</code> is automatically applied when <code>--useLogReg</code> is specified.

This option cannot be used in conjunction with [[#Fast Recalibration (--fast)|<code>--fast</code>]] and is overriden by <code>--fast</code>, but automatically applied by useLogReg

=== Fast Recalibration (<code>--fast</code>) ===

Use a compact representation of the Recalibration Table that only allows:
* at most 256 Read Groups
* maximum quality 63
* at most 127 cycles

This option will run faster than the default recalibration, but uses up to about 2.25G more memory than running without --fast.

This option cannot be used in conjunction with [[#Fit Model (--fitModel)|<code>--fitModel</code>]], or [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]] and overrides [[#Fit Model (--fitModel)|<code>--fitModel</code>]], but is overridden by [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]].

=== Allow Previous Base DBSNP (<code>--keepPrevDbsnp</code>) ===

By default bases where the previous base is in DBSNP are excluded from the Recalibration Table.

This option includes these bases in the building of the Recalibration Table.

=== Allow Previous Base Non-Match/Mismatch (<code>--keepPrevNonAdjacent</code>) ===

By default bases where the previous base is not a CIGAR Match/Mismatch are excluded from the Recalibration Table.

This option includes these bases in the building of the Recalibration Table.

=== Logistic Regression (<code>--useLogReg</code>) ===

Use the logistic regression empirical qualities for setting the new base qualities instead of the default formula.

This option automatically enables [[#Fit Model (--fitModel)|<code>--fitModel</code>]] and disables [[#Fast Recalibration (--fast)|<code>--fast</code>]].

=== Read the quality from a tag (<code>--qualField</code>) ===

If this parameter is set, then read the quality string from the specified tag name. If the tag is not found, the quality is read from the quality field.

=== Store the original quality (<code>--storeQualTag</code>) ===

If this parameter is set, the original quality will be stored as a string in the specified tag.

=== Skip Records with any of the Specified Flags (<code>--buildExcludeFlags</code>, <code>--applyExcludeFlags</code>) ===
Use <code>--buildExcludeFlags</code> to skip records with any of the specified flags set when building the recalibration table, default 0xF04.

By default, when building the recalibration table reads with any of the following flags set are skipped:
* unmapped
* secondary alignment
* fails QC checks
* duplicate
* supplementary alignment

Use <code>--applyExcludeFlags</code> to skip records with any of the specified flags set when applying the recalibration table. The default value is 0x000, do not skip any reads.

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: recab

2019-04-20T03:00:41Z

Mktrost: /* Handling Recalibration/Implementation Notes */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>recab</code> function of <code>[[bamUtil]]</code> =
The <code>recab</code> option of [[bamUtil]] recalibrates a SAM/BAM file.

Recalibration can also be called as an option of [[bamUtil: dedup]]. This will perform the recalibration and the deduping in the same set of steps, increasing processing speed.

==Handling Recalibration/Implementation Notes==

Recalibration is a 2-step process that loops through the file twice (stdin is not support as input):
# Build Recalibration Table
# Apply Recalibration Table

The Recalibration Table groups bases based on a set of covariates:
* Read Group
* Quality (either from the quality string or [[#Read the quality from a tag (--qualField)|from a tag]])
* Cycle (reverse complement for reverse strands)
* 1st/2nd read in pair
* Previous Cycle's Base (reverse complement for reverse strands)
* This Cycle's Base (reverse complement for reverse strands)

The Recalibration Table tracks the number of matches/mismatches for each set of covariates.

Only bases meeting all of the following criteria are used to Build the Recalibration Table:
* Read criteria
** not a duplicate
** mapped
** mapping quality != 0
** mapping quality != 255
* Base criteria
** match/mismatch (not an insertion/deletion/skip/clip)
** not a [[#DBSNP File (--dbsnp)|dbSNP position]]
** base quality > [[#Minimum Recalibration Base Quality (--minBaseQual)|minBaseQual (5 by default)]]
* Additional criteria for cycle != 1 (can be turned off via flags)
** previous base is a CIGAR Match/Mismatch (Use [[#Allow Previous Base Non-Match/Mismatch (--keepPrevNonAdjacent)|<code>--keepPrevNonAdjacent</code>]] to disable)
** previous base position is not a [[#DBSNP File (--dbsnp)|dbSNP position]] (Use [[#Allow Previous Base DBSNP (--keepPrevDbsnp)|<code>--keepPrevDbsnp</code>]] to disable)

The Recalibration Table is applied to all bases meeting all of the following criteria (even if they were not used for creating the table):
* base quality > [[#Minimum Recalibration Base Quality (--minBaseQual)|minBaseQual (5 by default)]]
* at least 1 match or mismatch for the set of covariates

Recalibrated Quality is: <math>-10 * \log \frac{mismatches + 1}{mismatches + matches + 1}</math>

Alternatively, [[#Logistic Regression (--useLogReg)|logistic regression]] can be used for calculating the new quality.

If the Recalibrated Quality is greater than [[#Maximum Recalibration Base Quality (--maxBaseQual)|maxBaseQual]], the updated quality is set to maxBaseQual.

Optionally, the previous quality can be [[#Store the original quality (--storeQualTag)|stored in a tag]].

The current recalibration logic was designed for recalibrating ILLUMINA data.

NOTE: GATK ignores/skips adapters, but our logic does not.

== How to use it ==

When <code>recab</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file ([[#input File (--in)|--in]]), the output SAM/BAM file ([[#output File (--out)|--out]]), and the reference file ([[#Reference File (--refFile)|--refFile]]) are required inputs.

Recommended usage with Deduper:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Deduper:

/usr/cluster/bin/bam recab --in ${INPUT}.bam --out ${OUTPUT}.bam --refFile ${REF} --dbsnp ${DBSNP} --storeQualTag OQ --maxBaseQual 40

= Usage =
./bam recab (options) --in <InputBamFile> --out <OutputFile> [--log <logFile>] [--verbose] [--noeof] [--params] --refFile <ReferenceFile> [--dbsnp <dbsnpFile>] [--minBaseQual <minBaseQual>] [--maxBaseQual <maxBaseQual>] [--blended <weight>] [--fitModel] [--fast] [--keepPrevDbsnp] [--keepPrevNonAdjacent] [--useLogReg] [--qualField <tag>] [--storeQualTag <tag>] [--buildExcludeFlags <flag>] [--applyExcludeFlags <flag>]

= Parameters =
<pre>
Required General Parameters :
--in <infile> : input BAM file name
--out <outfile> : output recalibration file name
Optional General Parameters :
--log <logfile> : log and summary statistics (default: [outfile].log)
--verbose : Turn on verbose mode
--noeof : do not expect an EOF block on a bam file.
--params : print the parameter settings

Recab Specific Required Parameters
--refFile <reference file> : reference file name
Recab Specific Optional Parameters :
--dbsnp <known variance file> : dbsnp file of positions
--minBaseQual <minBaseQual> : minimum base quality of bases to recalibrate (default: 5)
--maxBaseQual <maxBaseQual> : maximum recalibrated base quality (default: 50)
qualities over this value will be set to this value.
This setting is applied after binning (if applicable).
--blended <weight> : blended model weight
--fitModel : check if the logistic regression model fits the data
overriden by fast, but automatically applied by useLogReg
--fast : use a compact representation that only allows:
* at most 256 Read Groups
* maximum quality 63
* at most 127 cycles
overrides fitModel, but is overridden by useLogReg
uses up to about 2.25G more memory than running without --fast.
--keepPrevDbsnp : do not exclude entries where the previous base is in dbsnp when
building the recalibration table
By default they are excluded from the table.
--keepPrevNonAdjacent : do not exclude entries where the previous base is not adjacent
(not a Cigar M/X/=) when building the recalibration table
By default they are excluded from the table (except the first cycle).
--useLogReg : use logistic regression calculated quality for the new quality
automatically applies fitModel and overrides fast.
--qualField <quality tag> : tag to get the starting base quality
(default is to get it from the Quality field)
--storeQualTag <quality tag> : tag to store the previous quality into
--buildExcludeFlags <flag> : exclude reads with any of these flags set when building the
recalibration table. Default is 0xF04
--applyExcludeFlags <flag> : do not apply the recalibration table to any reads with any of these flags set
Quality Binning Parameters (optional):
Bin qualities by phred score, into the ranges specified by binQualS or binQualF (both cannot be used)
Ranges are specified by comma separated minimum phred score for the bin, example: 1,17,20,30,40,50,70
The first bin always starts at 0, so does not need to be specified.
By default, the bin value is the low end of the range.
--binQualS : Bin the Qualities as specified (phred): minQualOfBin2, minQualofBin3...
--binQualF : Bin the Qualities based on the specified file
--binMid : Use the mid point of the quality bin range for the quality value of the bin.
--binHigh : Use the high end of the quality bin range for the quality value of the bin.

</pre>
{{PhoneHomeParamDesc}}

== Required Generic Parameters ==
{{inBAMInputFile}}
{{outBAMOutputFile}}

== Optional Generic Parameters ==
=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

{{PhoneHomeParameters}}

== Required Recalibration Parameters ==
=== Reference File (<code>--refFile</code>) ===

The reference file is a required parameter used for comparing read bases to the reference.

== Optional Recalibration Parameters ==

=== DBSNP File (<code>--dbsnp</code>) ===

The dbsnp file that specifies positions to skip recalibrating. Tab delimited file with the chromosome in the first column and the 1-based position in the 2nd column.

=== Minimum Recalibration Base Quality (<code>--minBaseQual</code>) ===

When recalibrating reads, only positions with a base quality greater than this minimum phred quality will be recalibrated. If <code>--minBaseQual</code> is not specified, it is defaulted to 5.

The ILLUMINA specs indicate that any quality below 5 can be used as an error indicator so we do not want to recalibrate those.

=== Maximum Recalibration Base Quality (<code>--maxBaseQual</code>) ===

This value sets the maximum phred base quality assigned to a base after recalibrating. Any qualities above this value will be set to this value. It is defaulted to 50.

=== Blended Model Weight (<code>--blended</code>) ===

TBD - this parameter is not yet implemented.

=== Fit Model (<code>--fitModel</code>) ===

Check if the logistic regression model fits the data.

This option does NOT set the new qualities to the logistic regression calculated qualities, it only checks the fit. To apply the logistic regression qualities, see [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]]. <code>--fitModel</code> is automatically applied when <code>--useLogReg</code> is specified.

This option cannot be used in conjunction with [[#Fast Recalibration (--fast)|<code>--fast</code>]] and is overriden by <code>--fast</code>, but automatically applied by useLogReg

=== Fast Recalibration (<code>--fast</code>) ===

Use a compact representation of the Recalibration Table that only allows:
* at most 256 Read Groups
* maximum quality 63
* at most 127 cycles

This option will run faster than the default recalibration, but uses up to about 2.25G more memory than running without --fast.

This option cannot be used in conjunction with [[#Fit Model (--fitModel)|<code>--fitModel</code>]], or [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]] and overrides [[#Fit Model (--fitModel)|<code>--fitModel</code>]], but is overridden by [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]].

=== Allow Previous Base DBSNP (<code>--keepPrevDbsnp</code>) ===

By default bases where the previous base is in DBSNP are excluded from the Recalibration Table.

This option includes these bases in the building of the Recalibration Table.

=== Allow Previous Base Non-Match/Mismatch (<code>--keepPrevNonAdjacent</code>) ===

By default bases where the previous base is not a CIGAR Match/Mismatch are excluded from the Recalibration Table.

This option includes these bases in the building of the Recalibration Table.

=== Logistic Regression (<code>--useLogReg</code>) ===

Use the logistic regression empirical qualities for setting the new base qualities instead of the default formula.

This option automatically enables [[#Fit Model (--fitModel)|<code>--fitModel</code>]] and disables [[#Fast Recalibration (--fast)|<code>--fast</code>]].

=== Read the quality from a tag (<code>--qualField</code>) ===

If this parameter is set, then read the quality string from the specified tag name. If the tag is not found, the quality is read from the quality field.

=== Store the original quality (<code>--storeQualTag</code>) ===

If this parameter is set, the original quality will be stored as a string in the specified tag.

=== Skip Records with any of the Specified Flags (<code>--buildExcludeFlags</code>, <code>--applyExcludeFlags</code>) ===
Use <code>--buildExcludeFlags</code> to skip records with any of the specified flags set when building the recalibration table, default 0xF04.

By default, when building the recalibration table reads with any of the following flags set are skipped:
* unmapped
* secondary alignment
* fails QC checks
* duplicate
* supplementary alignment

Use <code>--applyExcludeFlags</code> to skip records with any of the specified flags set when applying the recalibration table. The default value is 0x000, do not skip any reads.

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: dedup

2019-04-09T02:31:36Z

Mktrost: /* Required Parameters */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>dedup</code> function of <code>[[bamUtil]]</code> =
The <code>dedup</code> option of [[bamUtil]] determines duplicates in a coordinate sorted SAM/BAM file. It either marks or removes the lower quality duplicates.

This tool also contains the option to perform recalibration.

NOTE: This tool does not properly work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.

NOTE: Dedup cannot read from stdin since it reads the input file twice.

Potential future features:
* Soft clip overlapping reads (for now, use: [[BamUtil: clipOverlap]])

==Handling Duplicates==

The deduper reads all the alignments in a coordinate-sorted SAM/BAM looking for duplicates, failing if the file is not coordinate-sorted.

The deduper assumes that duplicates in the input BAM file are not marked. When the deduper detects a marked duplicate in the input BAM file, it will throw an error and stop. To override this behavior, use the [[#Ignore Previous Duplicate Marking (--force)|<code>--force</code>]] option; in this mode, alignments that are marked as duplicates in the input file are unmarked before the deduper begins its detection algorithm. The result is that only duplicates detected by the deduper will be marked in or removed from the output file.

The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate. If a mate is not found at the expected position, an error message is printed (once per file) indicating this error. Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads. Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate. If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.

With the default settings this tool should produce similar results as Picard.

Use the [[#Treat Reads with Mates On Different Chromosomes As Single-Ended (--oneChrom)|<code>--oneChrom</code>]] option to treat reads with a mate on a different chromosome as single-ended. This option is useful if you are running the deduper on just a single chromosome. The code will use less memory with this option if mates are found on different chromosomes. (Picard does not specially handle mates on different chromosomes, so the --oneChrom option may produce different results than Picard.)

=== Implementation Notes ===
Duplicates are determined by checking for matching keys.

The Key is comprised of:
# Chromosome
# Orientation (forward/reverse)
# Unclipped Start(forward)/End(reverse)
# Library

Rules:
* Skip Unmapped Reads, they are not marked as duplicate
* Reads whose mate is unmapped are treated as single-end
* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:
*# A paired-end record has the same key (even if the pair is not proper/the mate is not found) -OR-
*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])
* Mark both Paired-End Reads Duplicate if:
# Another paired-end pair has the same set of keys and has a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])

This code assumes that at most 1000 bases are clipped at the start of a read.

Deduping requires two passes through the file, so cannot read from stdin.

==Handling Recalibration==
See [[BamUtil: recab]] for recalibration details.

Recalibration parameters can be applied to deduping when --recab is specified.

== How to use it ==

When <code>dedup</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file is required, [[#input File (--in)|input File (--in)]], and must be sorted by coordinate.

The output SAM/BAM file is also required, [[#output File (--out)|output File (--out)]].

Recommended usage with Recalibrator:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Recalibrator:

/usr/cluster/bin/bam dedup --in ${INPUT}.bam --out ${OUTPUT}.bam --force --oneChrom

= Usage =
./bam dedup --in <InputBamFile> --out <OutputBamFile> [--minQual <minPhred>] [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--excludeFlags <flag>] [--verbose] [--noeof] [--params] [--recab]

Additional Recalibration Usage is documented at [[BamUtil: recab#Usage|BamUtil: recab -> Usage]]

= Parameters =
<pre>
Required parameters :
--in <infile> : Input BAM file name (must be sorted)
--out <outfile> : Output BAM file name (same order with original file)
Optional parameters :
--minQual <int> : Only add scores over this phred quality when determining a read's quality (default: 15)
--log <logfile> : Log and summary statistics (default: [outfile].log, or stderr if --out starts with '-')
--oneChrom : Treat reads with mates on different chromosomes as single-ended.
--rmDups : Remove duplicates (default is to mark duplicates)
--force : Allow an already mark-duplicated BAM file, unmarking any previously marked
duplicates and apply this duplicate marking logic. Default is to throw errors
and exit when trying to run on an already mark-duplicated BAM
--excludeFlags <flag> : exclude reads with any of these flags set when determining or marking duplicates
by default (0xB04): exclude unmapped, secondary reads, QC failures, and supplementary reads
--verbose : Turn on verbose mode
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings
--recab : Recalibrate in addition to deduping
</pre>
{{PhoneHomeParamDesc}}

Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters|BamUtil: recab -> Parameters]]

== Required Parameters ==
{{inBAMInputFile|noStdin=1}}

Note: The input file must be sorted by coordinate.
{{outBAMOutputFile}}

== Optional Parameters==
=== Minimum Quality for Quality Calculations (<code>--minQual</code>) ===

When duplicate reads are encountered, the read with the highest quality is kept.

To determine the quality of a read, all of the phred base quality scores above the <code>--minQual</code> value are added together. If <code>--minQual</code> is not specified, it is defaulted to 15.

=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Treat Reads with Mates On Different Chromosomes As Single-Ended (<code>--oneChrom</code>) ===

If a read's mate is not found it will not be used for duplicate marking. If you are running on a single chromosome, all read's whose mates are on different chromosomes will not be used for duplicate marking. The <code>--oneChrom</code> option will treat reads with mates on a different chromosome as single-ended.

=== Remove Duplicates (<code>--rmDups</code>) ===

Instead of marking a read as duplicate in the flag, the <code>--rmDups</code> option will remove it from the output BAM file.

=== Ignore Previous Duplicate Marking (<code>--force</code>) ===

By default the deduper will throw an error and stop if a read is already marked as duplicate. The <code>--force</code> option will removes any previous duplicate marking and marks the reads from scratch. The resulting output file will only have reads determined by the deduper marked as duplicates.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0xB04

By default skips reads with any of the following flags set:
* unmapped
* secondary alignment
* fails QC checks
* supplementary reads

Secondary (0x100) and Supplementary (0x800) reads currently must be excluded.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

== Recalibrate (<code>--recab</code>) ==

This option will recalibrate the input file in addition to deduping.

See [[BamUtil: recab]] for recalibration details.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

Template:InBAMInputFile

2019-04-09T02:31:23Z

Mktrost:

{{{hdr|===}}} Input File ({{#if: {{{noParam|}}} | {{{noParam}}} argument|<code>{{{param|--in}}}</code>}}) {{{hdr|===}}}

{{#if: {{{noParam|}}} | The {{{noParam}}} argument is the name of the input SAM/BAM file. |
Use <code>{{{param|--in}}}</code> followed by your file name to specify the SAM/BAM input file.}}

The program automatically determines if your input file is SAM/BAM/uncompressed BAM without any input other than a filename from the user{{#if: {{{noStdin|}}} |.|, unless your input file is stdin.}}

{{#if: {{{noStdin|}}}|Note: This tool does not support input from stdin.|A <code>-</code> is used to indicate to read from stdin and the extension is used to determine the file type (no extension indicates SAM).}}

{|border="1" cellspacing="0" cellpadding="2"
|SAM/BAM/Uncompressed BAM from file
| <code>{{#if: {{{noParam|}}} | |--in}} yourFileName</code>
{{#if: {{{noStdin|}}} | |<tr>
<td>SAM from stdin</td>
<td><code>{{#if: {{{noParam|}}} | |--in}} -</code></td></tr>
<tr>
<td>BAM from stdin</td>
<td><code>{{#if: {{{noParam|}}} | |--in}} -.bam</code></td></tr>
<tr>
<td>Uncompressed BAM from stdin</td>
<td><code>{{#if: {{{noParam|}}} | |--in}} -.ubam</code></td></tr>}}
|}

Note: Uncompressed BAM is compressed using compression level-0 (so it is not an entirely uncompressed file). This matches the <code>samtools</code> implementation so pipes between our tools and <code>samtools</code> are supported.

Template:InBAMInputFile

2019-04-09T02:30:41Z

Mktrost:

{{{hdr|===}}} Input File ({{#if: {{{noParam|}}} | {{{noParam}}} argument|<code>{{{param|--in}}}</code>}}) {{{hdr|===}}}

{{#if: {{{noParam|}}} | The {{{noParam}}} argument is the name of the input SAM/BAM file. |
Use <code>{{{param|--in}}}</code> followed by your file name to specify the SAM/BAM input file.}}

The program automatically determines if your input file is SAM/BAM/uncompressed BAM without any input other than a filename from the user{{#if: {{{noStdin|}}} |.|, unless your input file is stdin.}}

{{#if: {{{noStdin|}}}|This tool does not support input from stdin.|A <code>-</code> is used to indicate to read from stdin and the extension is used to determine the file type (no extension indicates SAM).}}

{|border="1" cellspacing="0" cellpadding="2"
|SAM/BAM/Uncompressed BAM from file
| <code>{{#if: {{{noParam|}}} | |--in}} yourFileName</code>
{{#if: {{{noStdin|}}} | |<tr>
<td>SAM from stdin</td>
<td><code>{{#if: {{{noParam|}}} | |--in}} -</code></td></tr>
<tr>
<td>BAM from stdin</td>
<td><code>{{#if: {{{noParam|}}} | |--in}} -.bam</code></td></tr>
<tr>
<td>Uncompressed BAM from stdin</td>
<td><code>{{#if: {{{noParam|}}} | |--in}} -.ubam</code></td></tr>}}
|}

Note: Uncompressed BAM is compressed using compression level-0 (so it is not an entirely uncompressed file). This matches the <code>samtools</code> implementation so pipes between our tools and <code>samtools</code> are supported.

Template:InBAMInputFile

2019-04-09T02:29:34Z

Mktrost:

{{{hdr|===}}} Input File ({{#if: {{{noParam|}}} | {{{noParam}}} argument|<code>{{{param|--in}}}</code>}}) {{{hdr|===}}}

{{#if: {{{noParam|}}} | The {{{noParam}}} argument is the name of the input SAM/BAM file. |
Use <code>{{{param|--in}}}</code> followed by your file name to specify the SAM/BAM input file.}}

The program automatically determines if your input file is SAM/BAM/uncompressed BAM without any input other than a filename from the user{{#if: {{{noStdin|}}} .|, unless your input file is stdin.}}

{{#if: {{{noStdin|}}}|This tool does not support input from stdin.|A <code>-</code> is used to indicate to read from stdin and the extension is used to determine the file type (no extension indicates SAM).}}

{|border="1" cellspacing="0" cellpadding="2"
|SAM/BAM/Uncompressed BAM from file
| <code>{{#if: {{{noParam|}}} | |--in}} yourFileName</code>
{{#if: {{{noStdin|}}} | |<tr>
<td>SAM from stdin</td>
<td><code>{{#if: {{{noParam|}}} | |--in}} -</code></td></tr>
<tr>
<td>BAM from stdin</td>
<td><code>{{#if: {{{noParam|}}} | |--in}} -.bam</code></td></tr>
<tr>
<td>Uncompressed BAM from stdin</td>
<td><code>{{#if: {{{noParam|}}} | |--in}} -.ubam</code></td></tr>}}
|}

Note: Uncompressed BAM is compressed using compression level-0 (so it is not an entirely uncompressed file). This matches the <code>samtools</code> implementation so pipes between our tools and <code>samtools</code> are supported.

Template:InBAMInputFile

2019-04-09T02:19:06Z

Mktrost:

{{{hdr|===}}} Input File ({{#if: {{{noParam|}}} | {{{noParam}}} argument|<code>{{{param|--in}}}</code>}}) {{{hdr|===}}}

{{#if: {{{noParam|}}} | The {{{noParam}}} argument is the name of the input SAM/BAM file. |
Use <code>{{{param|--in}}}</code> followed by your file name to specify the SAM/BAM input file.}}

The program automatically determines if your input file is SAM/BAM/uncompressed BAM without any input other than a filename from the user{{#if: {{{noStdin|.}}}|, unless your input file is stdin.}}

{{#if: {{{noStdin|This tool does not support input from stdin.}}}|A <code>-</code> is used to indicate to read from stdin and the extension is used to determine the file type (no extension indicates SAM).}}

{|border="1" cellspacing="0" cellpadding="2"
|SAM/BAM/Uncompressed BAM from file
| <code>{{#if: {{{noParam|}}} | |--in}} yourFileName</code>
|-
|{{#if: {{{noStdin|}}} | |SAM from stdin}}
|{{#if: {{{noStdin|}}} | |<code>{{#if: {{{noParam|}}} | |--in}} -</code>}}
|-
|BAM from stdin
| <code>{{#if: {{{noParam|}}} | |--in}} -.bam</code>
|-
|Uncompressed BAM from stdin
| <code>{{#if: {{{noParam|}}} | |--in}} -.ubam</code>
|}

Note: Uncompressed BAM is compressed using compression level-0 (so it is not an entirely uncompressed file). This matches the <code>samtools</code> implementation so pipes between our tools and <code>samtools</code> are supported.

Template:InBAMInputFile

2019-04-09T02:17:30Z

Mktrost:

{{{hdr|===}}} Input File ({{#if: {{{noParam|}}} | {{{noParam}}} argument|<code>{{{param|--in}}}</code>}}) {{{hdr|===}}}

{{#if: {{{noParam|}}} | The {{{noParam}}} argument is the name of the input SAM/BAM file. |
Use <code>{{{param|--in}}}</code> followed by your file name to specify the SAM/BAM input file.}}

The program automatically determines if your input file is SAM/BAM/uncompressed BAM without any input other than a filename from the user{{#if: {{{noStdin|.}}}|, unless your input file is stdin.}}

{{#if: {{{noStdin|This tool does not support input from stdin.}}}|A <code>-</code> is used to indicate to read from stdin and the extension is used to determine the file type (no extension indicates SAM).}}

{|border="1" cellspacing="0" cellpadding="2"
|SAM/BAM/Uncompressed BAM from file
| <code>{{#if: {{{noParam|}}} | |--in}} yourFileName</code>
|-
|{{#if: {{{noStdin|}}} | |SAM from stdin}}
| <code>{{#if: {{{noParam|}}} | |--in}} -</code>
|-
|BAM from stdin
| <code>{{#if: {{{noParam|}}} | |--in}} -.bam</code>
|-
|Uncompressed BAM from stdin
| <code>{{#if: {{{noParam|}}} | |--in}} -.ubam</code>
|}

Note: Uncompressed BAM is compressed using compression level-0 (so it is not an entirely uncompressed file). This matches the <code>samtools</code> implementation so pipes between our tools and <code>samtools</code> are supported.

BamUtil: dedup

2019-04-09T02:03:08Z

Mktrost: /* Overview of the dedup function of bamUtil */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>dedup</code> function of <code>[[bamUtil]]</code> =
The <code>dedup</code> option of [[bamUtil]] determines duplicates in a coordinate sorted SAM/BAM file. It either marks or removes the lower quality duplicates.

This tool also contains the option to perform recalibration.

NOTE: This tool does not properly work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.

NOTE: Dedup cannot read from stdin since it reads the input file twice.

Potential future features:
* Soft clip overlapping reads (for now, use: [[BamUtil: clipOverlap]])

==Handling Duplicates==

The deduper reads all the alignments in a coordinate-sorted SAM/BAM looking for duplicates, failing if the file is not coordinate-sorted.

The deduper assumes that duplicates in the input BAM file are not marked. When the deduper detects a marked duplicate in the input BAM file, it will throw an error and stop. To override this behavior, use the [[#Ignore Previous Duplicate Marking (--force)|<code>--force</code>]] option; in this mode, alignments that are marked as duplicates in the input file are unmarked before the deduper begins its detection algorithm. The result is that only duplicates detected by the deduper will be marked in or removed from the output file.

The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate. If a mate is not found at the expected position, an error message is printed (once per file) indicating this error. Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads. Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate. If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.

With the default settings this tool should produce similar results as Picard.

Use the [[#Treat Reads with Mates On Different Chromosomes As Single-Ended (--oneChrom)|<code>--oneChrom</code>]] option to treat reads with a mate on a different chromosome as single-ended. This option is useful if you are running the deduper on just a single chromosome. The code will use less memory with this option if mates are found on different chromosomes. (Picard does not specially handle mates on different chromosomes, so the --oneChrom option may produce different results than Picard.)

=== Implementation Notes ===
Duplicates are determined by checking for matching keys.

The Key is comprised of:
# Chromosome
# Orientation (forward/reverse)
# Unclipped Start(forward)/End(reverse)
# Library

Rules:
* Skip Unmapped Reads, they are not marked as duplicate
* Reads whose mate is unmapped are treated as single-end
* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:
*# A paired-end record has the same key (even if the pair is not proper/the mate is not found) -OR-
*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])
* Mark both Paired-End Reads Duplicate if:
# Another paired-end pair has the same set of keys and has a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])

This code assumes that at most 1000 bases are clipped at the start of a read.

Deduping requires two passes through the file, so cannot read from stdin.

==Handling Recalibration==
See [[BamUtil: recab]] for recalibration details.

Recalibration parameters can be applied to deduping when --recab is specified.

== How to use it ==

When <code>dedup</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file is required, [[#input File (--in)|input File (--in)]], and must be sorted by coordinate.

The output SAM/BAM file is also required, [[#output File (--out)|output File (--out)]].

Recommended usage with Recalibrator:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Recalibrator:

/usr/cluster/bin/bam dedup --in ${INPUT}.bam --out ${OUTPUT}.bam --force --oneChrom

= Usage =
./bam dedup --in <InputBamFile> --out <OutputBamFile> [--minQual <minPhred>] [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--excludeFlags <flag>] [--verbose] [--noeof] [--params] [--recab]

Additional Recalibration Usage is documented at [[BamUtil: recab#Usage|BamUtil: recab -> Usage]]

= Parameters =
<pre>
Required parameters :
--in <infile> : Input BAM file name (must be sorted)
--out <outfile> : Output BAM file name (same order with original file)
Optional parameters :
--minQual <int> : Only add scores over this phred quality when determining a read's quality (default: 15)
--log <logfile> : Log and summary statistics (default: [outfile].log, or stderr if --out starts with '-')
--oneChrom : Treat reads with mates on different chromosomes as single-ended.
--rmDups : Remove duplicates (default is to mark duplicates)
--force : Allow an already mark-duplicated BAM file, unmarking any previously marked
duplicates and apply this duplicate marking logic. Default is to throw errors
and exit when trying to run on an already mark-duplicated BAM
--excludeFlags <flag> : exclude reads with any of these flags set when determining or marking duplicates
by default (0xB04): exclude unmapped, secondary reads, QC failures, and supplementary reads
--verbose : Turn on verbose mode
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings
--recab : Recalibrate in addition to deduping
</pre>
{{PhoneHomeParamDesc}}

Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters|BamUtil: recab -> Parameters]]

== Required Parameters ==
{{inBAMInputFile}}

Note: The input file must be sorted by coordinate.
{{outBAMOutputFile}}

== Optional Parameters==
=== Minimum Quality for Quality Calculations (<code>--minQual</code>) ===

When duplicate reads are encountered, the read with the highest quality is kept.

To determine the quality of a read, all of the phred base quality scores above the <code>--minQual</code> value are added together. If <code>--minQual</code> is not specified, it is defaulted to 15.

=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Treat Reads with Mates On Different Chromosomes As Single-Ended (<code>--oneChrom</code>) ===

If a read's mate is not found it will not be used for duplicate marking. If you are running on a single chromosome, all read's whose mates are on different chromosomes will not be used for duplicate marking. The <code>--oneChrom</code> option will treat reads with mates on a different chromosome as single-ended.

=== Remove Duplicates (<code>--rmDups</code>) ===

Instead of marking a read as duplicate in the flag, the <code>--rmDups</code> option will remove it from the output BAM file.

=== Ignore Previous Duplicate Marking (<code>--force</code>) ===

By default the deduper will throw an error and stop if a read is already marked as duplicate. The <code>--force</code> option will removes any previous duplicate marking and marks the reads from scratch. The resulting output file will only have reads determined by the deduper marked as duplicates.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0xB04

By default skips reads with any of the following flags set:
* unmapped
* secondary alignment
* fails QC checks
* supplementary reads

Secondary (0x100) and Supplementary (0x800) reads currently must be excluded.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

== Recalibrate (<code>--recab</code>) ==

This option will recalibrate the input file in addition to deduping.

See [[BamUtil: recab]] for recalibration details.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: dedup

2019-04-09T02:02:12Z

Mktrost: /* Implementation Notes */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>dedup</code> function of <code>[[bamUtil]]</code> =
The <code>dedup</code> option of [[bamUtil]] determines duplicates in a coordinate sorted SAM/BAM file. It either marks or removes the lower quality duplicates.

This tool also contains the option to perform recalibration.

NOTE: This tool does not properly work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.

Potential future features:
* Soft clip overlapping reads (for now, use: [[BamUtil: clipOverlap]])

==Handling Duplicates==

The deduper reads all the alignments in a coordinate-sorted SAM/BAM looking for duplicates, failing if the file is not coordinate-sorted.

The deduper assumes that duplicates in the input BAM file are not marked. When the deduper detects a marked duplicate in the input BAM file, it will throw an error and stop. To override this behavior, use the [[#Ignore Previous Duplicate Marking (--force)|<code>--force</code>]] option; in this mode, alignments that are marked as duplicates in the input file are unmarked before the deduper begins its detection algorithm. The result is that only duplicates detected by the deduper will be marked in or removed from the output file.

The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate. If a mate is not found at the expected position, an error message is printed (once per file) indicating this error. Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads. Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate. If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.

With the default settings this tool should produce similar results as Picard.

Use the [[#Treat Reads with Mates On Different Chromosomes As Single-Ended (--oneChrom)|<code>--oneChrom</code>]] option to treat reads with a mate on a different chromosome as single-ended. This option is useful if you are running the deduper on just a single chromosome. The code will use less memory with this option if mates are found on different chromosomes. (Picard does not specially handle mates on different chromosomes, so the --oneChrom option may produce different results than Picard.)

=== Implementation Notes ===
Duplicates are determined by checking for matching keys.

The Key is comprised of:
# Chromosome
# Orientation (forward/reverse)
# Unclipped Start(forward)/End(reverse)
# Library

Rules:
* Skip Unmapped Reads, they are not marked as duplicate
* Reads whose mate is unmapped are treated as single-end
* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:
*# A paired-end record has the same key (even if the pair is not proper/the mate is not found) -OR-
*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])
* Mark both Paired-End Reads Duplicate if:
# Another paired-end pair has the same set of keys and has a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])

This code assumes that at most 1000 bases are clipped at the start of a read.

Deduping requires two passes through the file, so cannot read from stdin.

==Handling Recalibration==
See [[BamUtil: recab]] for recalibration details.

Recalibration parameters can be applied to deduping when --recab is specified.

== How to use it ==

When <code>dedup</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file is required, [[#input File (--in)|input File (--in)]], and must be sorted by coordinate.

The output SAM/BAM file is also required, [[#output File (--out)|output File (--out)]].

Recommended usage with Recalibrator:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Recalibrator:

/usr/cluster/bin/bam dedup --in ${INPUT}.bam --out ${OUTPUT}.bam --force --oneChrom

= Usage =
./bam dedup --in <InputBamFile> --out <OutputBamFile> [--minQual <minPhred>] [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--excludeFlags <flag>] [--verbose] [--noeof] [--params] [--recab]

Additional Recalibration Usage is documented at [[BamUtil: recab#Usage|BamUtil: recab -> Usage]]

= Parameters =
<pre>
Required parameters :
--in <infile> : Input BAM file name (must be sorted)
--out <outfile> : Output BAM file name (same order with original file)
Optional parameters :
--minQual <int> : Only add scores over this phred quality when determining a read's quality (default: 15)
--log <logfile> : Log and summary statistics (default: [outfile].log, or stderr if --out starts with '-')
--oneChrom : Treat reads with mates on different chromosomes as single-ended.
--rmDups : Remove duplicates (default is to mark duplicates)
--force : Allow an already mark-duplicated BAM file, unmarking any previously marked
duplicates and apply this duplicate marking logic. Default is to throw errors
and exit when trying to run on an already mark-duplicated BAM
--excludeFlags <flag> : exclude reads with any of these flags set when determining or marking duplicates
by default (0xB04): exclude unmapped, secondary reads, QC failures, and supplementary reads
--verbose : Turn on verbose mode
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings
--recab : Recalibrate in addition to deduping
</pre>
{{PhoneHomeParamDesc}}

Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters|BamUtil: recab -> Parameters]]

== Required Parameters ==
{{inBAMInputFile}}

Note: The input file must be sorted by coordinate.
{{outBAMOutputFile}}

== Optional Parameters==
=== Minimum Quality for Quality Calculations (<code>--minQual</code>) ===

When duplicate reads are encountered, the read with the highest quality is kept.

To determine the quality of a read, all of the phred base quality scores above the <code>--minQual</code> value are added together. If <code>--minQual</code> is not specified, it is defaulted to 15.

=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Treat Reads with Mates On Different Chromosomes As Single-Ended (<code>--oneChrom</code>) ===

If a read's mate is not found it will not be used for duplicate marking. If you are running on a single chromosome, all read's whose mates are on different chromosomes will not be used for duplicate marking. The <code>--oneChrom</code> option will treat reads with mates on a different chromosome as single-ended.

=== Remove Duplicates (<code>--rmDups</code>) ===

Instead of marking a read as duplicate in the flag, the <code>--rmDups</code> option will remove it from the output BAM file.

=== Ignore Previous Duplicate Marking (<code>--force</code>) ===

By default the deduper will throw an error and stop if a read is already marked as duplicate. The <code>--force</code> option will removes any previous duplicate marking and marks the reads from scratch. The resulting output file will only have reads determined by the deduper marked as duplicates.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0xB04

By default skips reads with any of the following flags set:
* unmapped
* secondary alignment
* fails QC checks
* supplementary reads

Secondary (0x100) and Supplementary (0x800) reads currently must be excluded.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

== Recalibrate (<code>--recab</code>) ==

This option will recalibrate the input file in addition to deduping.

See [[BamUtil: recab]] for recalibration details.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil

2018-07-31T01:46:22Z

Mktrost:

[[Category:bamUtil]]
[[Category:C++]]
[[Category:Software]]

= bamUtil Overview =

bamUtil is a repository that contains several programs that perform operations on SAM/BAM files. All of these programs are built into a single executable, <code>bam</code>.

== Getting Help ==

If you have any questions please use the [http://groups.google.com/group/bamUtils bamUtil Google Group] to ask questions or recommend improvements to bamUtil.

Alternatively, you can e-mail me, Mary Kate Wing, at mktrost@umich.edu.

See [[BamUtil: FAQ]] to see if your question has already been answered.

== Where to Find It ==
{{ToolGitRepo|repoName=bamUtil}}

== Releases ==

If you prefer to run the last official release rather than the latest development version, you can download that here.

There are two versions of the release, one that include libStatGen and one that does not. If you already have libStatGen installed and want to use your own copy, use the version that does not include libStatGen.

=== Full Release (includes libStatGen) ===

To install an official release, unpack the downloaded file (tar xvf), cd into the bamUtil_x.x.x directory and type make all.

For version 1.0.14 and later, please download libStatGen and bamUtil separately:

'''Version 1.0.14 - Released 7/8/2015'''
*[[LibStatGen Download#Official Releases|libStatGen version 1.0.14]]
*[[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.14]]

'''Older Releases'''
* [[Media:BamUtilLibStatGen.1.0.13.tgz|BamUtilLibStatGen.1.0.13.tgz‎]] - Released 2/20/2015
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.13]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.13]] - see link for version updates

* [[Media:BamUtilLibStatGen.1.0.12.tar.gz|BamUtilLibStatGen.1.0.12.tgz‎]] - Released 5/14/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.12]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.12]] - see link for version updates
** Adds regions to [[BamUtil: mergeBam|mergeBam]]
** Accept ',' delimiters for the tags string input in [[BamUtil: squeeze|squeeze]], [[BamUtil: revert|revert]], & [[BamUtil: diff|diff]]

*[[Media:BamUtilLibStatGen.1.0.11.tar.gz|BamUtilLibStatGen.1.0.11.tar.gz‎]] - Released 2/28/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.11]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.11]] - see link for version updates
** Now properly supports 'B' & 'f' tags
** Cleanup - compile issues

*[[Media:BamUtilLibStatGen.1.0.10.tar.gz|BamUtilLibStatGen.1.0.10.tar.gz‎]] - Released 1/2/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.10]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.10]] - see link for version updates
** Adds PhoneHome/Version checking.

*[[Media:BamUtilLibStatGen.1.0.9.tgz|BamUtilLibStatGen.1.0.9.tgz‎]] - Released 7/7/2013
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.9]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.9]]
** Update to [[BamUtil: mergeBam|mergeBam]]
*** Update to ignore PG lines with duplicate IDs
*** Update to accept merges of matching RG lines
*** Update to log to stderr if no log/out file is specified
* There is no version 1.0.8. It was skipped to stay in line with libStatGen versions (libStatGen 1.0.8 added vcf support)
*[[Media:BamUtilLibStatGen.1.0.7.tgz|BamUtilLibStatGen.1.0.7.tgz‎]] - Released 1/29/2013
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.7]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.7]]
** Update to fix some compile issues on ubuntu 12.10
** Update use of SamRecord::getStringTag to expect the return of a const string pointer due to libStatGen v1.0.7 updates
** Update SamReferenceInfo usage due to libStatGen v1.0.7 updates
** Update to [[BamUtil: diff|diff]]
*** Fix DIFF to test and properly handle running out of available records. Previously no message was printed when this happened and there was a bug for which file it freed
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** Update to facilitate adding other overlap handling functions
** Update to [[BamUtil: mergeBam|mergeBam]] (formerly RGMergeBam)
*** Rename RGMergeBam to MergeBam
*** Update to handle files that already have an RG

*[[Media:BamUtilLibStatGen.1.0.6.tgz|BamUtilLibStatGen.1.0.6.tgz‎]] - Released 11/14/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.6]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.6]]
** Update to [[BamUtil: trimBam|trimBam]]
*** Update to allow trimming a different number of bases from each end of the read
*[[Media:BamUtilLibStatGen.1.0.5.tgz|BamUtilLibStatGen.1.0.5.tgz‎]] - Released 10/24/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.5]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.5]]
** Updates to: [[BamUtil: dedup|dedup]], [[BamUtil: polishBam|polishBam]], [[BamUtil: recab|recab]]
** Update to add compile option to compile without C++0x/C++11
** See [[#Release of just BamUtil (does not include libStatGen)|below]] for additional details on updates
*BamUtilLibStatGen.1.0.4.tgz‎ - Released skipped
*[[Media:BamUtilLibStatGen.1.0.3.tgz|BamUtilLibStatGen.1.0.3.tgz‎]] - Released 09/19/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.3]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.3]]
** Adds: [[BamUtil: dedup|dedup]] [[BamUtil: recab|recab]]
*[[Media:BamUtilLibStatGen.1.0.2.tgz|BamUtilLibStatGen.1.0.2.tgz‎]] - Released 05/16/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.2]]
** Adds: [[BamUtil: bam2FastQ|bam2FastQ]]
*[[Media:BamUtilLibStatGen.1.0.1.tgz|BamUtilLibStatGen.1.0.1.tgz‎]] - Released 05/04/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.1]]
** Adds: [[BamUtil: splitBam|splitBam]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: trimBam|trimBam]], [[BamUtil: polishBam|polishBam]], [[BamUtil: rgMergeBam|rgMergeBam]], [[BamUtil: gapInfo|gapInfo]]
** Adds additional functionality to [[BamUtil: stats|stats]]
** Adds leftShifting to [[BamUtil: writeRegion|writeRegion]] and [[BamUtil: convert|convert]]
** Adds more diff fields to [[BamUtil: diff|diff]]
* [[Media:BamUtilLibStatGen.1.0.0.tgz|BamUtilLibStatGen.1.0.0.tgz‎]] - Released 10/10/2011
**Initial release of bamUtil that includes libStatGen version 1.0.0. It started from the tool found in the deprecated StatGen repository.
**Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.0]] [[BamUtil: validate|validate]], [[BamUtil: convert|convert]], [[BamUtil: dumpHeader|dumpHeader]], [[BamUtil: splitChromosome|splitChromosome]], [[BamUtil: writeRegion|writeRegion]], [[BamUtil: dumpRefInfo|dumpRefInfo]], [[BamUtil: dumpIndex|dumpIndex]], [[BamUtil: readIndexedBam|readIndexedBam]], [[BamUtil: filter|filter]], [[BamUtil: readReference|readReference]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]], [[BamUtil: squeeze|squeeze]], [[BamUtil: findCigars|findCigars]], [[BamUtil: stats|stats]]

=== Release of just BamUtil (does not include libStatGen) ===

To install an official release, unpack the downloaded file (tar xvf), cd into the bamUtil_x.x.x directory and type make all.

'''BamUtil.1.0.14 Release Notes'''
* BamUtil Version 1.0.14 - Released 7/8/2015
** https://github.com/statgen/bamUtil/archive/v1.0.14.tar.gz
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.14]]
** Update [[BamUtil: trimBam|trimBam]]
*** Add option to soft clip (-c) instead of trimming
** Update [[BamUtil: clipOverlap|clipOverlap]]
*** Add option to mark reads as unmapped if they are entirely clipped
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** Add option to gzip the output files
*** Add option to split Read Groups into separate fastq files
*** Add option to get the quality from a tag
** Update [[BamUtil: recab|recab]]
*** Update to ignore ref 'N' when building the recalibration table
*** Add ability to bin
** Add Dedup_LowMem tool

'''Older Releases'''
* BamUtil Version 1.0.13 - Released 2/20/2015
** https://github.com/statgen/bamUtil/archive/v1.0.13.tar.gz
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.13]]
** Makefile Updates
*** Improve logic to determine actual path for the library
*** Update to append to USER_COMPILE_VARS even if specified on the command line
** Update [[BamUtil: writeRegion|writeRegion]]
*** Add option to specify readnames to keep in a file
*** Fixed bug that if a read overlapped 2 BED positions, it was printed twice
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** Update to skip non-primary reads
** Update to [[BamUtil: polishBam|polishBam]]
*** Update to handle '\t' string inputs and to add CO option
*** Fix MD5sum calculation to convert fasta to uppercase prior to calculating

* [[Media:BamUtil.1.0.12.tgz|BamUtil.1.0.12.tgz‎]] - Released 5/14/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.12]]
** Update [[BamUtil: mergeBam|mergeBam]]
*** Add a regions option
** Update to [[BamUtil: squeeze|squeeze]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]]
*** Also accept ',' instead of just ';' as the delimiter in the input tags string.

* [[Media:BamUtil.1.0.11.tgz|BamUtil.1.0.11.tgz‎]] - Released 2/28/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.11]]
*** Adds support for 'B' & 'f' tags that did not work properly before.
** Update [[BamUtil: splitBam|splitBam]] & [[BamUtil: polishBam|polishBam]]
*** Update to work properly if log & output file are not specified (no longer creates '.log')
** Update Main dummy/example tool to indicate the correct tool
** Update to [[BamUtil: bam2FastQ|bam2FastQ]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: filter|filter]], [[BamUtil: mergeBam|mergeBam]], [[BamUtil: splitBam|splitBam]], [[BamUtil: squeeze|squeeze]], [[BamUtil: stats|stats]]
*** Cleanup usage/parameter descriptions
** Update [[BamUtil: revert|revert]]
*** Update compatibility with libStatGen due to 'B' & 'f' tag handling updates
** Add tests for 'B' & 'f' tags

* [[Media:BamUtil.1.0.10.tar.gz|BamUtil.1.0.10.tar.gz‎]] - Released 1/2/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.10]]
** All
*** Add PhoneHome/version checking
*** Make sub-program names case independent
*** Fix Logger.cpp compiler warning
** Adds: [[BamUtil: explainFlags|explainFlags]] - describes the SAM/BAM flags based on the flag value
** Update to [[BamUtil: stats|stats]]
*** Fix Stats to not try to not try to process a record after it is out of the loop (it would already have been processed or is invalid)
** Update to [[BamUtil: splitBam|splitBam]]
*** fix description of --noeof option
** Update to [[BamUtil: writeRegion|writeRegion]]
*** add exclude/required flags
** Update to [[BamUtil: dedup|dedup]] & [[BamUtil: recab|recab]]
*** Ignore secondary reads for dedup and making the recalibration table.
*** skip QC Failures
*** add excludeFlags parameters
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** add exclude flags
*** fix bug for readName sorted when a read is filtered due to flags
*** add sorting validation
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** add --merge option to generate interleaved files.
*** update to open the input file before opening the output files, so if there is an error, the outputs aren't opened
** Update to [[BamUtil: mergeBam|mergeBam]]
*** add option to ignore the RG PI field when checking headers
*** add more informative header merge error messages

* [[Media:BamUtil.1.0.9.tgz|BamUtil.1.0.9.tgz‎]] - Released 7/7/2013
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.9]] (version 1.0.7 should also work)
** Update to [[BamUtil: mergeBam|mergeBam]]
*** Update to ignore PG lines with duplicate IDs
*** Update to accept merges of matching RG lines
*** Update to log to stderr if no log/out file is specified

*[[Media:BamUtil.1.0.7.tgz|BamUtil.1.0.7.tgz‎]] - Released 1/29/2013
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.7]] or above
** Update to fix some compile issues on ubuntu 12.10
** Update use of SamRecord::getStringTag to expect the return of a const string pointer due to libStatGen v1.0.7 updates
** Update SamReferenceInfo usage due to libStatGen v1.0.7 updates
** Update to [[BamUtil: diff|diff]]
*** Fix DIFF to test and properly handle running out of available records. Previously no message was printed when this happened and there was a bug for which file it freed
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** Update to facilitate adding other overlap handling functions
** Update to [[BamUtil: mergeBam|mergeBam]] (formerly RGMergeBam)
*** Rename RGMergeBam to MergeBam
*** Update to handle files that already have an RG
*[[Media:BamUtil.1.0.6.tgz|BamUtil.1.0.6.tgz‎]] - Released 11/14/2012
** Update to [[BamUtil: trimBam|trimBam]]
*** Update to allow trimming a different number of bases from each end of the read
*[[Media:BamUtil.1.0.5.tgz|BamUtil.1.0.5.tgz‎]] - Released 10/24/2012
** Update to [[BamUtil: dedup|dedup]]
*** Update logic for which pair to keep if they have the same quality
** Update to [[BamUtil: polishBam|polishBam]]
*** Update to print the number of successful header additions
** Update to [[BamUtil: recab|recab]]
*** Update to print the number of base skipped due to the base quality
** General Updates
*** Update to add compile option to compile without C++0x/C++11
*BamUtil.1.0.4.tgz‎ - Released skipped
*[[Media:BamUtil.1.0.3.tgz|BamUtil.1.0.3.tgz‎]] - Released 09/19/2012
** Adds: [[BamUtil: dedup|dedup]] [[BamUtil: recab|recab]]
** General Updates
*** Update Logger to write to stderr if output is stdout
** Update to [[BamUtil: stats|stats]]
*** Add required/exclude flags
*** Exclude Clips if excluding umapped
*** Add --withinRegion flag
*** Update phred/qual counts to be uint64_t instead of int to avoid overflow
** Update to [[BamUtil: validate|validate]]
*** Detect header failures
** Update to [[BamUtil: diff|diff]]
*** Update to specify chromosome/pos in ZP as a string rather than int so both can be shown
** Update to [[BamUtil: readReference|readReference]]
*** Output error message if the reference name is not found
** Update to [[BamUtil: splitChromosome|splitChromosome]]
*** Update to actually split the chromosomes and not just hard coded to output chromosomes ids 0-22
** Update Makefile to have cloneLib for cloning libStatGen
*[[Media:BamUtil.1.0.2.tgz|BamUtil.1.0.2.tgz‎]] - Released 05/16/2012
** Adds: [[BamUtil: bam2FastQ|bam2FastQ]]
*[[Media:BamUtil.1.0.1.tgz|BamUtil.1.0.1.tgz‎]] - Released 05/04/2012
** Adds: [[BamUtil: splitBam|splitBam]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: trimBam|trimBam]], [[BamUtil: polishBam|polishBam]], [[BamUtil: rgMergeBam|rgMergeBam]], [[BamUtil: gapInfo|gapInfo]]
** Adds additional functionality to [[BamUtil: stats|stats]]
** Adds leftShifting to [[BamUtil: writeRegion|writeRegion]] and [[BamUtil: convert|convert]]
** Adds more diff fields to [[BamUtil: diff|diff]]
*[[Media:BamUtil.1.0.0.tgz|BamUtil.1.0.0.tgz‎]] - Released 10/10/2011
**Initial release of just bamUtil. It started from the tool found in the deprecated StatGen repository.
**Contains: [[BamUtil: validate|validate]], [[BamUtil: convert|convert]], [[BamUtil: dumpHeader|dumpHeader]], [[BamUtil: splitChromosome|splitChromosome]], [[BamUtil: writeRegion|writeRegion]], [[BamUtil: dumpRefInfo|dumpRefInfo]], [[BamUtil: dumpIndex|dumpIndex]], [[BamUtil: readIndexedBam|readIndexedBam]], [[BamUtil: filter|filter]], [[BamUtil: readReference|readReference]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]], [[BamUtil: squeeze|squeeze]], [[BamUtil: findCigars|findCigars]], [[BamUtil: stats|stats]]

== Citation ==
If you use BamUtil, please cite our publication on GotCloud which includes BamUtil:
[http://genome.cshlp.org/content/early/2015/04/14/gr.176552.114.abstract Jun, Goo, et al. "An efficient and scalable analysis framework for variant extraction and refinement from population scale DNA sequence data." Genome research (2015): gr-176552.]

= Programs =

The software reads the beginning of an input file to determine if it is SAM/BAM. To determine the format (SAM/BAM) of the output file, the software checks the output file's extension. If the extension is ".bam" it writes a BAM file, otherwise it writes a SAM file.

{{BamUtilPrograms}}

LibStatGen Troubleshooting

2017-11-13T04:20:15Z

Mktrost: /* Possible Compile Errors */

= Possible Compile Errors =

'''If compilation fails due to warnings being treated as errors, please contact us so we can fix the warnings. As a work-around to get it to compile, you can disable the treatment of warnings as errors by editing libStatGen/general/Makefile to remove -Werror.'''

The following are a list of compile errors that have been encountered with recommendations on what they mean and how to fix them.

== GenomeSequence.cpp:161: error: explicit instantiation... ==
If you are compiling on OSX and encounter this error:
;GenomeSequence.cpp<nowiki>:</nowiki> In instantiation of ‘std<nowiki>::</nowiki>basic_ostream<_CharT, _Traits>& std<nowiki>::</nowiki>operator<<(std<nowiki>::</nowiki>basic_ostream<_CharT, _Traits>&, const std<nowiki>::</nowiki>basic_string<_CharT, _Traits, _Alloc>&) [with _CharT = char, _Traits = std<nowiki>::</nowiki>char_traits<char>, _Alloc = std<nowiki>::</nowiki>allocator<char>]’<nowiki>:</nowiki> GenomeSequence.cpp<nowiki>:</nowiki>161<nowiki>:</nowiki> instantiated from here GenomeSequence.cpp<nowiki>:</nowiki>161<nowiki>:</nowiki> error<nowiki>:</nowiki> explicit instantiation of ‘std<nowiki>::</nowiki>basic_ostream<_CharT, _Traits>& std<nowiki>::</nowiki>operator<<(std<nowiki>::</nowiki>basic_ostream<_CharT, _Traits>&, const std<nowiki>::</nowiki>basic_string<_CharT, _Traits, _Alloc>&) [with _CharT = char, _Traits = std<nowiki>::</nowiki>char_traits<char>, _Alloc = std<nowiki>::</nowiki>allocator<char>]’ but no definition available
: Try compiling with:
make USER_COMPILE_VARS=-mmacosx-version-min=10.8

== error: Sam[filename].h: No such file or directory or "'Sam[classname]' has not been declared" ==
This error and other, similar errors with "Sam....h: No such file or directory" or "'Sam...' has not been declared" typically means that the library build failed. Check for a previous error in your compile.

A successful compile of the library will publish SamFile.h and all the other header files from the library into libStatGen/include. If one of the programs within the statgen repository fails to compile with one of these errors, then the library did not successfully publish the header files and did not successfully build.

== InputFile.h (or any other library file): No such file or directory ==
Errors like the following:
<pre>
FastQFile.cpp:20:23: InputFile.h: No such file or directory
In file included from FastQFile.cpp:21:
FastQFile.h:23:26: StringBasics.h: No such file or directory
</pre>
Tend to occur if the library directory containing the file has not yet been compiled. Try recompiling the entire library by typing make in libStatGen.

== Compile Error on first time building ==
Record the error.

Try to recompile and see if that fixes the problem. If it does fix the compile error, let me know what the original error was so I can fix the dependency so it compiles in the correct order.

'''NOTE: In the latest version, this should be fixed.'''

Template:ToolGitRepo

2017-11-13T04:18:27Z

Mktrost:

{{#if: {{{noDownload|}}} | | The {{{repoName}}} repository is available both via [[#Releases|release downloads]] and via github.}}

On github, https://github.com/statgen/{{{repoName}}}, you can both browse and download the {{{repoName}}} source code as well as explore the history of changes.

You can obtain the source either with or without git.

{{#if: {{{nolibStatGen|}}} | |
{{#if: {{{libStatGen|}}} | A copy of libStatGen is included in certain releases of some statgen tools. |
The releases may be available both with and without libStatGen included.

If you do not use the release version that already contains libStatGen, you need to download the library: [[C++ Library: libStatGen|libStatGen]].

If you try to compile {{{repoName}}} and it cannot find libStatGen, it will fail and provide instructions of what to do next:
* if libStatGen is in a different location then expected
** follow the directions to set the path to libStatGen
* if libStatGen is not downloaded and you have git
** <code>make libStatGen</code> will download via git and build libStatGen
* if libStatGen is not downloaded and you don't have git
** See [[C++ Library: libStatGen|libStatGen]]
}}
}}

=== Using Git To Track the Current Development Version ===

==== Clone (get your own copy) ====
You can create your own git clone (copy) using:
git clone https://github.com/statgen/{{{repoName}}}.git
or
git clone git://github.com/statgen/{{{repoName}}}.git

Either of these commands create a directory called <code>{{{repoName}}}</code> in the current directory.

Then just <code>cd {{{repoName}}}</code> and [[#Building|compile]].

==== Get the latest Updates (update your copy) ====
To update your copy to the latest version (a major advantage of using git):
# <code>cd pathToYourCopy/{{{repoName}}}</code>
# <code>make clean</code>
# <code>git pull</code>
# <code>make all</code>

==== Git Refresher ====
If you decide to use git, but need a refresher, see [[How To Use Git]] or [https://statgen.sph.umich.edu/wiki/How_To_Use_Git Notes on how to use git] (if you have access)

=== Downloading From GitHub Without Git ===
If you download the latest code/version, make sure you periodically update it by downloading a newer version.

From github you can download:
# Latest Code (master branch)
#: via Website
#:# Goto: https://github.com/statgen/{{{repoName}}}
#:# Click on the <code>Download ZIP</code> button on the right side panel.
#: via Command Line
#:: <code>wget https://github.com/statgen/{{{repoName}}}/archive/master.tar.gz</code>
#::: or
#:: <code>wget https://github.com/statgen/{{{repoName}}}/archive/master.zip</code>
# Specific Release (via a tag)
#: via Website
#:# Goto: https://github.com/statgen/{{{repoName}}}/releases to see the available releases
#:# Click <code>zip</code> or <code>tar.gz</code> for the desired version.
#: via Command Line
#:: <code>wget https://github.com/statgen/{{{repoName}}}/archive/<tagName>.tar.gz</code>
#::: or
#:: <code>wget https://github.com/statgen/{{{repoName}}}/archive/<tagName>.zip</code>
{{#if: {{{latestReleaseBranch|}}} |
#Latest Release (via a branch)
#: via Website
#:# Goto: https://github.com/statgen/{{{repoName}}}/tree/{{{latestReleaseBranch}}} to see the latest release
#:# Click on the <code>Download ZIP</code> button on the right side panel.
#: via Command Line
#:: <code>wget https://github.com/statgen/{{{repoName}}}/archive/{{{latestReleaseBranch}}}.tar.gz</code>
#::: or
#:: <code>wget https://github.com/statgen/{{{repoName}}}/archive/{{{latestReleaseBranch}}}.zip</code>
}}

After downloading the file, uncompress (unzip/untar) it. The directory created will be named <code>{{{repoName}}}-<name of version you downloaded></code>.

{{#if: {{{skipBuild|}}}||
== Building ==
After obtaining the {{{repoName}}} repository (either by download or from github), compile the code using:
<code>make all</code>

Object (.o) files are compiled into the <code>obj</code> directory with a subdirectory <code>debug</code> and <code>profile</code> for the debugging and profiling objects.

{{#if: {{{libBaseName|}}}|
This creates the libraries, {{{libBaseName}}}.a, {{{libBaseName}}}_debug.a, {{{libBaseName}}}_profile.a at the top level directory.
|
This creates the executable(s) in the <code>{{{repoName}}}/bin/</code> directory, the debug executable(s) in the <code>{{{repoName}}}/bin/debug/</code> directory, and the profiling executable(s) in the <code>{{{repoName}}}/bin/profile/</code> directory.

<code>make install</code> installs the opt binary if you have permission.
}}

<code>make test</code> compiles for opt, debug, and profile and runs the tests (found in the <code>test</code> subdirectory).

To see all make options, type <code>make help</code>.
}}

If compilation fails due to warnings being treated as errors, please contact us so we can fix the warnings. As a work-around to get it to compile, you can disable the treatment of warnings as errors by editing libStatGen/general/Makefile to remove <code>-Werror</code>.

LibStatGen Download

2017-11-13T04:04:10Z

Mktrost: /* Official Releases */

= General libStatGen Information =
See [[C++ Library: libStatGen]].

= Latest Development Versions =
For information about obtaining the latest Development Versions, see [[C++ Library: libStatGen#Where to Find It|Where to Find libStatGen]]

= Official Releases =
If you prefer to run the last official release rather than the latest development version, you can download that here.

To install an official release, unpack the downloaded file (tar xvf), cd into the libStatGen directory and type <code>make all</code>.

'''LibStatGen.1.0.14 Release Notes'''
*Version 1.0.14 - Released 7/8/2015
** https://github.com/statgen/libStatGen/archive/v1.0.14.tar.gz
** Contains classes for general processing (string, reference file, parameters), bam, glf, fastq, and vcf
*** General Updates:
**** Compiling Updates:
***** PedigreeLoader.cpp - fix bool vs int definition
***** InputFile - add 'const' to isOpen and ifeof methods. Also add comments about usage of 'operator >>'
*** Bam/Tabix Updates:
**** Update how the index is stored in memory to not preallocate the maximum number of bins for each reference as this could use a lot of memory if there are a lot of references
*** BAM Updates:
**** SamFilter
***** Update to clear MapQ and N/A flags when marking as unmapped
***** Fix bug in clipping logic that didn't properly adjust the start position if the read was already clipped
*** VCF Updates:
**** Add missing include for c++11

'''LibStatGen.1.0.13 Release Notes'''
*Version 1.0.13 - Released 2/20/2015
** https://github.com/statgen/libStatGen/archive/v1.0.13.tar.gz
** Contains classes for general processing (string, reference file, parameters), bam, glf, fastq, and vcf
*** General Updates:
**** Added knetfile support to InputFile (only for bgzf files if KNET_ON=1 is set on compile line
**** GzipHeader - fix signed/unsigned comparison compile issue
**** Upgrade bgzf (also allow append mode)
*** Makefiles Updates:
**** Update dependencies to fix/improve some compiling issues
*** PhoneHome Updates:
**** Allow PhoneHome to be a legacy parameter
**** Update new version available message to stand out more
**** Fix bugs in checks for new version since it wasn't finding it.
*** BAM Updates:
**** SamReferenceInfo operator== - fix to not check hash, since hash could be different for same info
**** Pileup - improve error message for buffer overrun
*** VCF Updates:
**** Add option to update GT field
**** Add subset samples parameter to ReadRecord
**** Improve handling of adding CO header

''' Older Releases'''
*[[Media:LibStatGen.1.0.12.tgz‎|LibStatGen.1.0.12.tgz‎]] - Released 5/14/2014
** Contains classes for general processing (string, reference file, parameters), bam, glf, fastq, and vcf
*** General Updates:
**** Improve Makefiles for adding additional libraries/creating external libraries
**** Cleanup compiling for mingw
**** Cleanup Copyrights
*** BAM Updates:
**** Update rmTags & getTagsString to accept ',' delimiter rather than just ';'
*[[Media:LibStatGen.1.0.11.tgz‎|LibStatGen.1.0.11.tar.gz‎]] - Released 2/28/2014
** Contains classes for general processing (string, reference file, parameters), bam, glf, fastq, and vcf
*** General Updates:
**** Cleanup compile warnings - should improve compatibility with Windows, RedHat, and older and newer compilers
**** StringBasics: Improve off-by-one overflow handling (from github user wavexx)
*** BAM Updates:
**** Fix it to properly handle 'B' & 'f' tags
**** Fix handling of duplicate tags
*** PhoneHome Updates:
**** Add parameters names (not values) to PhoneHome URL
*** VCF Updates:
**** Move allele count logic from VcfFileReader into VcfRecord.
*[[Media:LibStatGen.1.0.10.tar.gz‎|LibStatGen.1.0.10.tar.gz‎]] - Released 1/2/2014
** Contains classes for general processing (string, reference file, parameters), bam, glf, fastq, and vcf
*** Updates:
**** Add PhoneHome/version checking logic
***General Updates:
**** fix use of strncpy to fix compile problem for some users
**** Fix compile problem on gcc 4.8.1 by adding -Wno-strict-overflow in general/Makefile after -Werror
**** Reduce the default IFILE read buffer size from 1MB to 64KB.
**** Fix ifclose to take a IFILE reference so it can make the pointer 0 after closing
**** StringBasics::AsInteger now returns false if the string is empty.
**** GzipFileType - update to accept -.gz as stdin/stdout
**** Remove unnecssary line from Parameters Status call
**** Fix memory leak on Tabix destructor
*** Vcf Updates:
**** VCF - add class/methods to discard records based on ID
*** Bam Updates:
**** SamFlag - add isSecondary check
**** Fix SamFile so it only allows '-', '-.sam', '-.bam', or '-.ubam' to indicate reading from stdin rather than anything that starts with '-' in order to protect it from misinterpreting another parameter as the stdin filename
**** SamRecord: Add warnings about Duplicate Tags being overwritten
**** SamFile - improve sort order validation error messages and compare using both String::Compare and strcmp to handle various sort mechanisms since neither matches samtools exactly (if either is successful, the sort order is OK)
*** Glf Updates:
**** GlfFile cleanup bugs/handle missind endMarkers & update "get" methods to be const

*[[Media:LibStatGen.1.0.9.tgz‎|LibStatGen.1.0.9.tgz‎]] - Released 7/7/2013
** Contains classes for general processing (string, reference file, parameters), bam, glf, fastq, and vcf
*** Updates:
**** Update Makefile processing to check for a Makefile in the test directory before recursively calling make in there
**** Replace CFLAGS wtih COMPFLAGS
**** Update the Windows WIN32 if statements
***General Updates:
**** Add boundary check to StringBasics memory
**** Added support for new CIGAR characters (X & =)
**** Update bug in PedigreeLoader to specify WHITESPACE
**** add accessors for getting tabix information
*** Vcf Updates:
**** add ability to drop trailing missing Genotype fields
**** check that the tabix index is for VCF file type
**** Fix broken dependency
*** Bam Updates:
**** SamHeaderRecord: add accessor to get the value for the key
**** update some classes to have virtual destructors

*[[Media:LibStatGen.1.0.8.tgz‎|LibStatGen.1.0.8.tgz‎]] - Released 1/29/2013
** Contains classes for general processing (string, reference file, parameters), bam, glf, fastq, and vcf
*** Updates:
**** Added VCF file handling (merged the vcf branch to master)
**** Add support for reading Tabix files
**** Add ReusableVector class that holds vectors that will remove only the last or be reset & reused without clearing memory.

*[[Media:LibStatGen.1.0.7.tgz‎|LibStatGen.1.0.7.tgz‎]] - Released 1/29/2013
** Contains classes for general processing (string, reference file, parameters), bam, glf, and fastq
*** Updates:
**** Add operator== to StringIntHash
**** Cleanup compiling for files that are just headers with no .cpp
**** Update Makefile TOOLOBJ to handle fortran
**** ReferenceSequence.h fix compile problem on upgraded system
**** Add Makefile.extlib for compiling libraries that use libStatGen
*** BAM Updates:
**** Update to allow copying SamHeaderRecords and automatically update the SQ ReferenceInfo
**** Update SamFileHeader::getReferenceInfo() to return a SamReferenceInfo reference rather than ptr
**** Update SamRecord::getStringTag to return a const string pointer instead of just a string pointer
**** Update SamCoordOutput to add flushAll method

* [[Media:LibStatGen.1.0.6.tgz‎|LibStatGen.1.0.6.tgz‎]] - Released 11/14/2012
** Contains classes for general processing (string, reference file, parameters), bam, glf, and fastq
*** Updates:
**** Fix a bug in the make install command

* [[Media:LibStatGen.1.0.5.tgz‎|LibStatGen.1.0.5.tgz‎]] - Released 10/24/2012
** Contains classes for general processing (string, reference file, parameters), bam, glf, and fastq
*** Updates:
**** Cleanup compile warnings
**** Support compiling without ZLIB (use: make ZLIB_AVAIL=0)
*** BAM Updates:
**** Update to add error messages about failing to add a header
**** Handle a vtype of 'B' in the TAG field (treats it like a string)

* LibStatGen.1.0.4.tgz‎ - Released skipped
*[[Media:LibStatGen.1.0.3.tgz‎|LibStatGen.1.0.3.tgz‎]] - Released 09/19/2012
** Contains classes for general processing (string, reference file, parameters), bam, glf, and fastq
*** Additions:
**** Fix Makefile COMPILE_ANY_CHANGE logic to actually work
**** Fix compile problems on macs
***** Add #include <cstddef> to GzipFileType.h & #include <unistd.h> to MemoryMapArray.h
*** General Additions:
**** Update DBSNP processing in GenomeSequence to also accept gzipped files
**** Add support of fasta files in GenomeSequence
**** Update GenomeSequence constructor/open to create the umfa version if it doesn't already exist
**** Update StringAlias to add getAliases
**** Update StringHash to have an option to do case sensitive (instead of always case-insensitive)
**** Make library building PHONY so it is always checked for updates and only updated if objects are newer
*** BAM Additions:
**** Update BamIndex hex constant to use ULL
**** Cleanup handling of header errors
**** Update SamFileHeader to use case sensitive hashes
**** Make repeated header line keys a warning, not an error
*** FASTQ Additions:
**** Update to output average qualities
*[[Media:LibStatGen.1.0.2.tgz‎|LibStatGen.1.0.2.tgz‎]] - Released 05/16/2012
** Contains classes for general processing (string, reference file, parameters), bam, glf, and fastq
** Additions:
*** Add reverseComplement method to BaseUtilities.h
*[[Media:LibStatGen.1.0.1.tgz‎|LibStatGen.1.0.1.tgz‎]] - Released 05/04/2012
** Contains classes for general processing (string, reference file, parameters), bam, glf, and fastq
** BAM Additions:
*** Ability to use require/exclude flags when reading
*** CigarHelper for soft clipping
*** Fixed bug in BamIndex that wasn't closing the file
*** Additional cleanup/updates, see https://github.com/statgen/libStatGen/commits/master/bam for all changes between 10/10/2011 and 05/04/2012.
** General Additions:
*** Modifications to improve file handling/string processing
*** Fixed some compile warnings
*** Update Parameters class to always print warnings about unknown parameters to stderr
*** Update PedigreePerson to create/delete the strings object
*** Cleanup GenomeSequence to remove code that is also in BaseAsciiMap
** Samtools Modifications:
*** Do not use knetfile and fix compile warning

*[[Media:LibStatGen.1.0.0.tgz‎|LibStatGen.1.0.0.tgz‎]] - Released 10/10/2011
** Contains classes for general processiong (string, reference file, parameters), bam, glf, and fastq
** Initial release of libStatGen. It started from the library found in the deprecated StatGen repository.

BamUtil

2017-11-13T03:55:56Z

Mktrost: /* Full Release (includes libStatGen) */

[[Category:bamUtil]]
[[Category:C++]]
[[Category:Software]]

= bamUtil Overview =

bamUtil is a repository that contains several programs that perform operations on SAM/BAM files. All of these programs are built into a single executable, <code>bam</code>.

== Getting Help ==

If you have any questions please use the [http://groups.google.com/group/bamUtils bamUtil Google Group] to ask questions or recommend improvements to bamUtil.

Alternatively, you can e-mail me, Mary Kate Wing, at mktrost@umich.edu.

See [[BamUtil: FAQ]] to see if your question has already been answered.

== Where to Find It ==
{{ToolGitRepo|repoName=bamUtil}}

== Releases ==

If you prefer to run the last official release rather than the latest development version, you can download that here.

There are two versions of the release, one that include libStatGen and one that does not. If you already have libStatGen installed and want to use your own copy, use the version that does not include libStatGen.

=== Full Release (includes libStatGen) ===

To install an official release, unpack the downloaded file (tar xvf), cd into the bamUtil_x.x.x directory and type make all.

For version 1.0.14 and later, please download libStatGen and bamUtil separately:

'''Version 1.0.14 - Released 7/8/2015'''
*[[LibStatGen Download#Official Releases|libStatGen version 1.0.14]]
*[[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.14]]

'''Older Releases'''
* [[Media:BamUtilLibStatGen.1.0.13.tgz|BamUtilLibStatGen.1.0.13.tgz‎]] - Released 2/20/2015
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.13]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.13]] - see link for version updates

* [[Media:BamUtilLibStatGen.1.0.12.tar.gz|BamUtilLibStatGen.1.0.12.tgz‎]] - Released 5/14/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.12]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.12]] - see link for version updates
** Adds regions to [[BamUtil: mergeBam|mergeBam]]
** Accept ',' delimiters for the tags string input in [[BamUtil: squeeze|squeeze]], [[BamUtil: revert|revert]], & [[BamUtil: diff|diff]]

*[[Media:BamUtilLibStatGen.1.0.11.tar.gz|BamUtilLibStatGen.1.0.11.tar.gz‎]] - Released 2/28/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.11]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.11]] - see link for version updates
** Now properly supports 'B' & 'f' tags
** Cleanup - compile issues

*[[Media:BamUtilLibStatGen.1.0.10.tar.gz|BamUtilLibStatGen.1.0.10.tar.gz‎]] - Released 1/2/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.10]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.10]] - see link for version updates
** Adds PhoneHome/Version checking.

*[[Media:BamUtilLibStatGen.1.0.9.tgz|BamUtilLibStatGen.1.0.9.tgz‎]] - Released 7/7/2013
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.9]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.9]]
** Update to [[BamUtil: mergeBam|mergeBam]]
*** Update to ignore PG lines with duplicate IDs
*** Update to accept merges of matching RG lines
*** Update to log to stderr if no log/out file is specified
* There is no version 1.0.8. It was skipped to stay in line with libStatGen versions (libStatGen 1.0.8 added vcf support)
*[[Media:BamUtilLibStatGen.1.0.7.tgz|BamUtilLibStatGen.1.0.7.tgz‎]] - Released 1/29/2013
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.7]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.7]]
** Update to fix some compile issues on ubuntu 12.10
** Update use of SamRecord::getStringTag to expect the return of a const string pointer due to libStatGen v1.0.7 updates
** Update SamReferenceInfo usage due to libStatGen v1.0.7 updates
** Update to [[BamUtil: diff|diff]]
*** Fix DIFF to test and properly handle running out of available records. Previously no message was printed when this happened and there was a bug for which file it freed
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** Update to facilitate adding other overlap handling functions
** Update to [[BamUtil: mergeBam|mergeBam]] (formerly RGMergeBam)
*** Rename RGMergeBam to MergeBam
*** Update to handle files that already have an RG

*[[Media:BamUtilLibStatGen.1.0.6.tgz|BamUtilLibStatGen.1.0.6.tgz‎]] - Released 11/14/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.6]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.6]]
** Update to [[BamUtil: trimBam|trimBam]]
*** Update to allow trimming a different number of bases from each end of the read
*[[Media:BamUtilLibStatGen.1.0.5.tgz|BamUtilLibStatGen.1.0.5.tgz‎]] - Released 10/24/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.5]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.5]]
** Updates to: [[BamUtil: dedup|dedup]], [[BamUtil: polishBam|polishBam]], [[BamUtil: recab|recab]]
** Update to add compile option to compile without C++0x/C++11
** See [[#Release of just BamUtil (does not include libStatGen)|below]] for additional details on updates
*BamUtilLibStatGen.1.0.4.tgz‎ - Released skipped
*[[Media:BamUtilLibStatGen.1.0.3.tgz|BamUtilLibStatGen.1.0.3.tgz‎]] - Released 09/19/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.3]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.3]]
** Adds: [[BamUtil: dedup|dedup]] [[BamUtil: recab|recab]]
*[[Media:BamUtilLibStatGen.1.0.2.tgz|BamUtilLibStatGen.1.0.2.tgz‎]] - Released 05/16/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.2]]
** Adds: [[BamUtil: bam2FastQ|bam2FastQ]]
*[[Media:BamUtilLibStatGen.1.0.1.tgz|BamUtilLibStatGen.1.0.1.tgz‎]] - Released 05/04/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.1]]
** Adds: [[BamUtil: splitBam|splitBam]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: trimBam|trimBam]], [[BamUtil: polishBam|polishBam]], [[BamUtil: rgMergeBam|rgMergeBam]], [[BamUtil: gapInfo|gapInfo]]
** Adds additional functionality to [[BamUtil: stats|stats]]
** Adds leftShifting to [[BamUtil: writeRegion|writeRegion]] and [[BamUtil: convert|convert]]
** Adds more diff fields to [[BamUtil: diff|diff]]
* [[Media:BamUtilLibStatGen.1.0.0.tgz|BamUtilLibStatGen.1.0.0.tgz‎]] - Released 10/10/2011
**Initial release of bamUtil that includes libStatGen version 1.0.0. It started from the tool found in the deprecated StatGen repository.
**Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.0]] [[BamUtil: validate|validate]], [[BamUtil: convert|convert]], [[BamUtil: dumpHeader|dumpHeader]], [[BamUtil: splitChromosome|splitChromosome]], [[BamUtil: writeRegion|writeRegion]], [[BamUtil: dumpRefInfo|dumpRefInfo]], [[BamUtil: dumpIndex|dumpIndex]], [[BamUtil: readIndexedBam|readIndexedBam]], [[BamUtil: filter|filter]], [[BamUtil: readReference|readReference]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]], [[BamUtil: squeeze|squeeze]], [[BamUtil: findCigars|findCigars]], [[BamUtil: stats|stats]]

=== Release of just BamUtil (does not include libStatGen) ===

To install an official release, unpack the downloaded file (tar xvf), cd into the bamUtil_x.x.x directory and type make all.

'''BamUtil.1.0.14 Release Notes'''
* BamUtil Version 1.0.14 - Released 7/8/2015
** https://github.com/statgen/bamUtil/archive/v1.0.14.tar.gz
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.14]]
** Update [[BamUtil: trimBam|trimBam]]
*** Add option to soft clip (-c) instead of trimming
** Update [[BamUtil: clipOverlap|clipOverlap]]
*** Add option to mark reads as unmapped if they are entirely clipped
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** Add option to gzip the output files
*** Add option to split Read Groups into separate fastq files
*** Add option to get the quality from a tag
** Update [[BamUtil: recab|recab]]
*** Update to ignore ref 'N' when building the recalibration table
*** Add ability to bin
** Add Dedup_LowMem tool

'''Older Releases'''
* BamUtil Version 1.0.13 - Released 2/20/2015
** https://github.com/statgen/bamUtil/archive/v1.0.13.tar.gz
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.13]]
** Makefile Updates
*** Improve logic to determine actual path for the library
*** Update to append to USER_COMPILE_VARS even if specified on the command line
** Update [[BamUtil: writeRegion|writeRegion]]
*** Add option to specify readnames to keep in a file
*** Fixed bug that if a read overlapped 2 BED positions, it was printed twice
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** Update to skip non-primary reads
** Update to [[BamUtil: polishBam|polishBam]]
*** Update to handle '\t' string inputs and to add CO option
*** Fix MD5sum calculation to convert fasta to uppercase prior to calculating

* [[Media:BamUtil.1.0.12.tgz|BamUtil.1.0.12.tgz‎]] - Released 5/14/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.12]]
** Update [[BamUtil: mergeBam|mergeBam]]
*** Add a regions option
** Update to [[BamUtil: squeeze|squeeze]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]]
*** Also accept ',' instead of just ';' as the delimiter in the input tags string.

* [[Media:BamUtil.1.0.11.tgz|BamUtil.1.0.11.tgz‎]] - Released 2/28/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.11]]
*** Adds support for 'B' & 'f' tags that did not work properly before.
** Update [[BamUtil: splitBam|splitBam]] & [[BamUtil: polishBam|polishBam]]
*** Update to work properly if log & output file are not specified (no longer creates '.log')
** Update Main dummy/example tool to indicate the correct tool
** Update to [[BamUtil: bam2FastQ|bam2FastQ]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: filter|filter]], [[BamUtil: mergeBam|mergeBam]], [[BamUtil: splitBam|splitBam]], [[BamUtil: squeeze|squeeze]], [[BamUtil: stats|stats]]
*** Cleanup usage/parameter descriptions
** Update [[BamUtil: revert|revert]]
*** Update compatibility with libStatGen due to 'B' & 'f' tag handling updates
** Add tests for 'B' & 'f' tags

* [[Media:BamUtil.1.0.10.tar.gz|BamUtil.1.0.10.tar.gz‎]] - Released 1/2/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.10]]
** All
*** Add PhoneHome/version checking
*** Make sub-program names case independent
*** Fix Logger.cpp compiler warning
** Adds: [[BamUtil: explainFlags|explainFlags]] - describes the SAM/BAM flags based on the flag value
** Update to [[BamUtil: stats|stats]]
*** Fix Stats to not try to not try to process a record after it is out of the loop (it would already have been processed or is invalid)
** Update to [[BamUtil: splitBam|splitBam]]
*** fix description of --noeof option
** Update to [[BamUtil: writeRegion|writeRegion]]
*** add exclude/required flags
** Update to [[BamUtil: dedup|dedup]] & [[BamUtil: recab|recab]]
*** Ignore secondary reads for dedup and making the recalibration table.
*** skip QC Failures
*** add excludeFlags parameters
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** add exclude flags
*** fix bug for readName sorted when a read is filtered due to flags
*** add sorting validation
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** add --merge option to generate interleaved files.
*** update to open the input file before opening the output files, so if there is an error, the outputs aren't opened
** Update to [[BamUtil: mergeBam|mergeBam]]
*** add option to ignore the RG PI field when checking headers
*** add more informative header merge error messages

* [[Media:BamUtil.1.0.9.tgz|BamUtil.1.0.9.tgz‎]] - Released 7/7/2013
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.9]] (version 1.0.7 should also work)
** Update to [[BamUtil: mergeBam|mergeBam]]
*** Update to ignore PG lines with duplicate IDs
*** Update to accept merges of matching RG lines
*** Update to log to stderr if no log/out file is specified

*[[Media:BamUtil.1.0.7.tgz|BamUtil.1.0.7.tgz‎]] - Released 1/29/2013
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.7]] or above
** Update to fix some compile issues on ubuntu 12.10
** Update use of SamRecord::getStringTag to expect the return of a const string pointer due to libStatGen v1.0.7 updates
** Update SamReferenceInfo usage due to libStatGen v1.0.7 updates
** Update to [[BamUtil: diff|diff]]
*** Fix DIFF to test and properly handle running out of available records. Previously no message was printed when this happened and there was a bug for which file it freed
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** Update to facilitate adding other overlap handling functions
** Update to [[BamUtil: mergeBam|mergeBam]] (formerly RGMergeBam)
*** Rename RGMergeBam to MergeBam
*** Update to handle files that already have an RG
*[[Media:BamUtil.1.0.6.tgz|BamUtil.1.0.6.tgz‎]] - Released 11/14/2012
** Update to [[BamUtil: trimBam|trimBam]]
*** Update to allow trimming a different number of bases from each end of the read
*[[Media:BamUtil.1.0.5.tgz|BamUtil.1.0.5.tgz‎]] - Released 10/24/2012
** Update to [[BamUtil: dedup|dedup]]
*** Update logic for which pair to keep if they have the same quality
** Update to [[BamUtil: polishBam|polishBam]]
*** Update to print the number of successful header additions
** Update to [[BamUtil: recab|recab]]
*** Update to print the number of base skipped due to the base quality
** General Updates
*** Update to add compile option to compile without C++0x/C++11
*BamUtil.1.0.4.tgz‎ - Released skipped
*[[Media:BamUtil.1.0.3.tgz|BamUtil.1.0.3.tgz‎]] - Released 09/19/2012
** Adds: [[BamUtil: dedup|dedup]] [[BamUtil: recab|recab]]
** General Updates
*** Update Logger to write to stderr if output is stdout
** Update to [[BamUtil: stats|stats]]
*** Add required/exclude flags
*** Exclude Clips if excluding umapped
*** Add --withinRegion flag
*** Update phred/qual counts to be uint64_t instead of int to avoid overflow
** Update to [[BamUtil: validate|validate]]
*** Detect header failures
** Update to [[BamUtil: diff|diff]]
*** Update to specify chromosome/pos in ZP as a string rather than int so both can be shown
** Update to [[BamUtil: readReference|readReference]]
*** Output error message if the reference name is not found
** Update to [[BamUtil: splitChromosome|splitChromosome]]
*** Update to actually split the chromosomes and not just hard coded to output chromosomes ids 0-22
** Update Makefile to have cloneLib for cloning libStatGen
*[[Media:BamUtil.1.0.2.tgz|BamUtil.1.0.2.tgz‎]] - Released 05/16/2012
** Adds: [[BamUtil: bam2FastQ|bam2FastQ]]
*[[Media:BamUtil.1.0.1.tgz|BamUtil.1.0.1.tgz‎]] - Released 05/04/2012
** Adds: [[BamUtil: splitBam|splitBam]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: trimBam|trimBam]], [[BamUtil: polishBam|polishBam]], [[BamUtil: rgMergeBam|rgMergeBam]], [[BamUtil: gapInfo|gapInfo]]
** Adds additional functionality to [[BamUtil: stats|stats]]
** Adds leftShifting to [[BamUtil: writeRegion|writeRegion]] and [[BamUtil: convert|convert]]
** Adds more diff fields to [[BamUtil: diff|diff]]
*[[Media:BamUtil.1.0.0.tgz|BamUtil.1.0.0.tgz‎]] - Released 10/10/2011
**Initial release of just bamUtil. It started from the tool found in the deprecated StatGen repository.
**Contains: [[BamUtil: validate|validate]], [[BamUtil: convert|convert]], [[BamUtil: dumpHeader|dumpHeader]], [[BamUtil: splitChromosome|splitChromosome]], [[BamUtil: writeRegion|writeRegion]], [[BamUtil: dumpRefInfo|dumpRefInfo]], [[BamUtil: dumpIndex|dumpIndex]], [[BamUtil: readIndexedBam|readIndexedBam]], [[BamUtil: filter|filter]], [[BamUtil: readReference|readReference]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]], [[BamUtil: squeeze|squeeze]], [[BamUtil: findCigars|findCigars]], [[BamUtil: stats|stats]]

= Programs =

The software reads the beginning of an input file to determine if it is SAM/BAM. To determine the format (SAM/BAM) of the output file, the software checks the output file's extension. If the extension is ".bam" it writes a BAM file, otherwise it writes a SAM file.

{{BamUtilPrograms}}

BamUtil

2017-11-13T03:49:26Z

Mktrost: /* Release of just BamUtil (does not include libStatGen) */

[[Category:bamUtil]]
[[Category:C++]]
[[Category:Software]]

= bamUtil Overview =

bamUtil is a repository that contains several programs that perform operations on SAM/BAM files. All of these programs are built into a single executable, <code>bam</code>.

== Getting Help ==

If you have any questions please use the [http://groups.google.com/group/bamUtils bamUtil Google Group] to ask questions or recommend improvements to bamUtil.

Alternatively, you can e-mail me, Mary Kate Wing, at mktrost@umich.edu.

See [[BamUtil: FAQ]] to see if your question has already been answered.

== Where to Find It ==
{{ToolGitRepo|repoName=bamUtil}}

== Releases ==

If you prefer to run the last official release rather than the latest development version, you can download that here.

There are two versions of the release, one that include libStatGen and one that does not. If you already have libStatGen installed and want to use your own copy, use the version that does not include libStatGen.

=== Full Release (includes libStatGen) ===

To install an official release, unpack the downloaded file (tar xvf), cd into the bamUtil_x.x.x directory and type make all.

[[Media:BamUtilLibStatGen.1.0.13.tgz|BamUtilLibStatGen.1.0.13.tgz‎]] - Released 2/20/2015

'''BamUtilLibStatGen.1.0.13 Release Notes'''
* Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.13]] - see link for version updates
* Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.13]] - see link for version updates

'''Older Releases'''
* [[Media:BamUtilLibStatGen.1.0.12.tar.gz|BamUtilLibStatGen.1.0.12.tgz‎]] - Released 5/14/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.12]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.12]] - see link for version updates
** Adds regions to [[BamUtil: mergeBam|mergeBam]]
** Accept ',' delimiters for the tags string input in [[BamUtil: squeeze|squeeze]], [[BamUtil: revert|revert]], & [[BamUtil: diff|diff]]

*[[Media:BamUtilLibStatGen.1.0.11.tar.gz|BamUtilLibStatGen.1.0.11.tar.gz‎]] - Released 2/28/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.11]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.11]] - see link for version updates
** Now properly supports 'B' & 'f' tags
** Cleanup - compile issues

*[[Media:BamUtilLibStatGen.1.0.10.tar.gz|BamUtilLibStatGen.1.0.10.tar.gz‎]] - Released 1/2/2014
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.10]] - see link for version updates
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.10]] - see link for version updates
** Adds PhoneHome/Version checking.

*[[Media:BamUtilLibStatGen.1.0.9.tgz|BamUtilLibStatGen.1.0.9.tgz‎]] - Released 7/7/2013
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.9]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.9]]
** Update to [[BamUtil: mergeBam|mergeBam]]
*** Update to ignore PG lines with duplicate IDs
*** Update to accept merges of matching RG lines
*** Update to log to stderr if no log/out file is specified
* There is no version 1.0.8. It was skipped to stay in line with libStatGen versions (libStatGen 1.0.8 added vcf support)
*[[Media:BamUtilLibStatGen.1.0.7.tgz|BamUtilLibStatGen.1.0.7.tgz‎]] - Released 1/29/2013
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.7]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.7]]
** Update to fix some compile issues on ubuntu 12.10
** Update use of SamRecord::getStringTag to expect the return of a const string pointer due to libStatGen v1.0.7 updates
** Update SamReferenceInfo usage due to libStatGen v1.0.7 updates
** Update to [[BamUtil: diff|diff]]
*** Fix DIFF to test and properly handle running out of available records. Previously no message was printed when this happened and there was a bug for which file it freed
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** Update to facilitate adding other overlap handling functions
** Update to [[BamUtil: mergeBam|mergeBam]] (formerly RGMergeBam)
*** Rename RGMergeBam to MergeBam
*** Update to handle files that already have an RG

*[[Media:BamUtilLibStatGen.1.0.6.tgz|BamUtilLibStatGen.1.0.6.tgz‎]] - Released 11/14/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.6]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.6]]
** Update to [[BamUtil: trimBam|trimBam]]
*** Update to allow trimming a different number of bases from each end of the read
*[[Media:BamUtilLibStatGen.1.0.5.tgz|BamUtilLibStatGen.1.0.5.tgz‎]] - Released 10/24/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.5]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.5]]
** Updates to: [[BamUtil: dedup|dedup]], [[BamUtil: polishBam|polishBam]], [[BamUtil: recab|recab]]
** Update to add compile option to compile without C++0x/C++11
** See [[#Release of just BamUtil (does not include libStatGen)|below]] for additional details on updates
*BamUtilLibStatGen.1.0.4.tgz‎ - Released skipped
*[[Media:BamUtilLibStatGen.1.0.3.tgz|BamUtilLibStatGen.1.0.3.tgz‎]] - Released 09/19/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.3]]
** Contains: [[#Release of just BamUtil (does not include libStatGen)|bamUtil version 1.0.3]]
** Adds: [[BamUtil: dedup|dedup]] [[BamUtil: recab|recab]]
*[[Media:BamUtilLibStatGen.1.0.2.tgz|BamUtilLibStatGen.1.0.2.tgz‎]] - Released 05/16/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.2]]
** Adds: [[BamUtil: bam2FastQ|bam2FastQ]]
*[[Media:BamUtilLibStatGen.1.0.1.tgz|BamUtilLibStatGen.1.0.1.tgz‎]] - Released 05/04/2012
** Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.1]]
** Adds: [[BamUtil: splitBam|splitBam]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: trimBam|trimBam]], [[BamUtil: polishBam|polishBam]], [[BamUtil: rgMergeBam|rgMergeBam]], [[BamUtil: gapInfo|gapInfo]]
** Adds additional functionality to [[BamUtil: stats|stats]]
** Adds leftShifting to [[BamUtil: writeRegion|writeRegion]] and [[BamUtil: convert|convert]]
** Adds more diff fields to [[BamUtil: diff|diff]]
* [[Media:BamUtilLibStatGen.1.0.0.tgz|BamUtilLibStatGen.1.0.0.tgz‎]] - Released 10/10/2011
**Initial release of bamUtil that includes libStatGen version 1.0.0. It started from the tool found in the deprecated StatGen repository.
**Contains: [[LibStatGen Download#Official Releases|libStatGen version 1.0.0]] [[BamUtil: validate|validate]], [[BamUtil: convert|convert]], [[BamUtil: dumpHeader|dumpHeader]], [[BamUtil: splitChromosome|splitChromosome]], [[BamUtil: writeRegion|writeRegion]], [[BamUtil: dumpRefInfo|dumpRefInfo]], [[BamUtil: dumpIndex|dumpIndex]], [[BamUtil: readIndexedBam|readIndexedBam]], [[BamUtil: filter|filter]], [[BamUtil: readReference|readReference]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]], [[BamUtil: squeeze|squeeze]], [[BamUtil: findCigars|findCigars]], [[BamUtil: stats|stats]]

=== Release of just BamUtil (does not include libStatGen) ===

To install an official release, unpack the downloaded file (tar xvf), cd into the bamUtil_x.x.x directory and type make all.

'''BamUtil.1.0.14 Release Notes'''
* BamUtil Version 1.0.14 - Released 7/8/2015
** https://github.com/statgen/bamUtil/archive/v1.0.14.tar.gz
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.14]]
** Update [[BamUtil: trimBam|trimBam]]
*** Add option to soft clip (-c) instead of trimming
** Update [[BamUtil: clipOverlap|clipOverlap]]
*** Add option to mark reads as unmapped if they are entirely clipped
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** Add option to gzip the output files
*** Add option to split Read Groups into separate fastq files
*** Add option to get the quality from a tag
** Update [[BamUtil: recab|recab]]
*** Update to ignore ref 'N' when building the recalibration table
*** Add ability to bin
** Add Dedup_LowMem tool

'''Older Releases'''
* BamUtil Version 1.0.13 - Released 2/20/2015
** https://github.com/statgen/bamUtil/archive/v1.0.13.tar.gz
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.13]]
** Makefile Updates
*** Improve logic to determine actual path for the library
*** Update to append to USER_COMPILE_VARS even if specified on the command line
** Update [[BamUtil: writeRegion|writeRegion]]
*** Add option to specify readnames to keep in a file
*** Fixed bug that if a read overlapped 2 BED positions, it was printed twice
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** Update to skip non-primary reads
** Update to [[BamUtil: polishBam|polishBam]]
*** Update to handle '\t' string inputs and to add CO option
*** Fix MD5sum calculation to convert fasta to uppercase prior to calculating

* [[Media:BamUtil.1.0.12.tgz|BamUtil.1.0.12.tgz‎]] - Released 5/14/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.12]]
** Update [[BamUtil: mergeBam|mergeBam]]
*** Add a regions option
** Update to [[BamUtil: squeeze|squeeze]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]]
*** Also accept ',' instead of just ';' as the delimiter in the input tags string.

* [[Media:BamUtil.1.0.11.tgz|BamUtil.1.0.11.tgz‎]] - Released 2/28/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.11]]
*** Adds support for 'B' & 'f' tags that did not work properly before.
** Update [[BamUtil: splitBam|splitBam]] & [[BamUtil: polishBam|polishBam]]
*** Update to work properly if log & output file are not specified (no longer creates '.log')
** Update Main dummy/example tool to indicate the correct tool
** Update to [[BamUtil: bam2FastQ|bam2FastQ]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: filter|filter]], [[BamUtil: mergeBam|mergeBam]], [[BamUtil: splitBam|splitBam]], [[BamUtil: squeeze|squeeze]], [[BamUtil: stats|stats]]
*** Cleanup usage/parameter descriptions
** Update [[BamUtil: revert|revert]]
*** Update compatibility with libStatGen due to 'B' & 'f' tag handling updates
** Add tests for 'B' & 'f' tags

* [[Media:BamUtil.1.0.10.tar.gz|BamUtil.1.0.10.tar.gz‎]] - Released 1/2/2014
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.10]]
** All
*** Add PhoneHome/version checking
*** Make sub-program names case independent
*** Fix Logger.cpp compiler warning
** Adds: [[BamUtil: explainFlags|explainFlags]] - describes the SAM/BAM flags based on the flag value
** Update to [[BamUtil: stats|stats]]
*** Fix Stats to not try to not try to process a record after it is out of the loop (it would already have been processed or is invalid)
** Update to [[BamUtil: splitBam|splitBam]]
*** fix description of --noeof option
** Update to [[BamUtil: writeRegion|writeRegion]]
*** add exclude/required flags
** Update to [[BamUtil: dedup|dedup]] & [[BamUtil: recab|recab]]
*** Ignore secondary reads for dedup and making the recalibration table.
*** skip QC Failures
*** add excludeFlags parameters
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** add exclude flags
*** fix bug for readName sorted when a read is filtered due to flags
*** add sorting validation
** Update to [[BamUtil: bam2FastQ|bam2FastQ]]
*** add --merge option to generate interleaved files.
*** update to open the input file before opening the output files, so if there is an error, the outputs aren't opened
** Update to [[BamUtil: mergeBam|mergeBam]]
*** add option to ignore the RG PI field when checking headers
*** add more informative header merge error messages

* [[Media:BamUtil.1.0.9.tgz|BamUtil.1.0.9.tgz‎]] - Released 7/7/2013
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.9]] (version 1.0.7 should also work)
** Update to [[BamUtil: mergeBam|mergeBam]]
*** Update to ignore PG lines with duplicate IDs
*** Update to accept merges of matching RG lines
*** Update to log to stderr if no log/out file is specified

*[[Media:BamUtil.1.0.7.tgz|BamUtil.1.0.7.tgz‎]] - Released 1/29/2013
** Requires, but does not include: [[LibStatGen Download#Official Releases|libStatGen version 1.0.7]] or above
** Update to fix some compile issues on ubuntu 12.10
** Update use of SamRecord::getStringTag to expect the return of a const string pointer due to libStatGen v1.0.7 updates
** Update SamReferenceInfo usage due to libStatGen v1.0.7 updates
** Update to [[BamUtil: diff|diff]]
*** Fix DIFF to test and properly handle running out of available records. Previously no message was printed when this happened and there was a bug for which file it freed
** Update to [[BamUtil: clipOverlap|clipOverlap]]
*** Update to facilitate adding other overlap handling functions
** Update to [[BamUtil: mergeBam|mergeBam]] (formerly RGMergeBam)
*** Rename RGMergeBam to MergeBam
*** Update to handle files that already have an RG
*[[Media:BamUtil.1.0.6.tgz|BamUtil.1.0.6.tgz‎]] - Released 11/14/2012
** Update to [[BamUtil: trimBam|trimBam]]
*** Update to allow trimming a different number of bases from each end of the read
*[[Media:BamUtil.1.0.5.tgz|BamUtil.1.0.5.tgz‎]] - Released 10/24/2012
** Update to [[BamUtil: dedup|dedup]]
*** Update logic for which pair to keep if they have the same quality
** Update to [[BamUtil: polishBam|polishBam]]
*** Update to print the number of successful header additions
** Update to [[BamUtil: recab|recab]]
*** Update to print the number of base skipped due to the base quality
** General Updates
*** Update to add compile option to compile without C++0x/C++11
*BamUtil.1.0.4.tgz‎ - Released skipped
*[[Media:BamUtil.1.0.3.tgz|BamUtil.1.0.3.tgz‎]] - Released 09/19/2012
** Adds: [[BamUtil: dedup|dedup]] [[BamUtil: recab|recab]]
** General Updates
*** Update Logger to write to stderr if output is stdout
** Update to [[BamUtil: stats|stats]]
*** Add required/exclude flags
*** Exclude Clips if excluding umapped
*** Add --withinRegion flag
*** Update phred/qual counts to be uint64_t instead of int to avoid overflow
** Update to [[BamUtil: validate|validate]]
*** Detect header failures
** Update to [[BamUtil: diff|diff]]
*** Update to specify chromosome/pos in ZP as a string rather than int so both can be shown
** Update to [[BamUtil: readReference|readReference]]
*** Output error message if the reference name is not found
** Update to [[BamUtil: splitChromosome|splitChromosome]]
*** Update to actually split the chromosomes and not just hard coded to output chromosomes ids 0-22
** Update Makefile to have cloneLib for cloning libStatGen
*[[Media:BamUtil.1.0.2.tgz|BamUtil.1.0.2.tgz‎]] - Released 05/16/2012
** Adds: [[BamUtil: bam2FastQ|bam2FastQ]]
*[[Media:BamUtil.1.0.1.tgz|BamUtil.1.0.1.tgz‎]] - Released 05/04/2012
** Adds: [[BamUtil: splitBam|splitBam]], [[BamUtil: clipOverlap|clipOverlap]], [[BamUtil: trimBam|trimBam]], [[BamUtil: polishBam|polishBam]], [[BamUtil: rgMergeBam|rgMergeBam]], [[BamUtil: gapInfo|gapInfo]]
** Adds additional functionality to [[BamUtil: stats|stats]]
** Adds leftShifting to [[BamUtil: writeRegion|writeRegion]] and [[BamUtil: convert|convert]]
** Adds more diff fields to [[BamUtil: diff|diff]]
*[[Media:BamUtil.1.0.0.tgz|BamUtil.1.0.0.tgz‎]] - Released 10/10/2011
**Initial release of just bamUtil. It started from the tool found in the deprecated StatGen repository.
**Contains: [[BamUtil: validate|validate]], [[BamUtil: convert|convert]], [[BamUtil: dumpHeader|dumpHeader]], [[BamUtil: splitChromosome|splitChromosome]], [[BamUtil: writeRegion|writeRegion]], [[BamUtil: dumpRefInfo|dumpRefInfo]], [[BamUtil: dumpIndex|dumpIndex]], [[BamUtil: readIndexedBam|readIndexedBam]], [[BamUtil: filter|filter]], [[BamUtil: readReference|readReference]], [[BamUtil: revert|revert]], [[BamUtil: diff|diff]], [[BamUtil: squeeze|squeeze]], [[BamUtil: findCigars|findCigars]], [[BamUtil: stats|stats]]

= Programs =

The software reads the beginning of an input file to determine if it is SAM/BAM. To determine the format (SAM/BAM) of the output file, the software checks the output file's extension. If the extension is ".bam" it writes a BAM file, otherwise it writes a SAM file.

{{BamUtilPrograms}}

BamUtil: trimBam

2017-11-13T03:28:36Z

Mktrost: /* Soft Clipping Notes (--clip/-c) */

= Overview of the <code>trimBam</code> function of <code>bamUtil</code> =
The <code>trimBam</code> option on the [[bamUtil]] executable trims the end of reads in a SAM/BAM file, changing read ends to ‘N’ and quality to ‘!’, or by soft clipping (if command-line option, <code>--clip</code> is specified).

= Usage =
./bam trimBam [inFile] [outFile] [num-bases-to-trim-on-each-side]

Version 1.0.6 and later:

Alternately, the number of bases from each side can be specified (either or both -L/-R (--left/--right) can be specified):
./bam trimBam [inFile] [outFile] -L [num-bases-to-trim-from-left] -R [num-bases-to-trim-from-right]

By default reverse strands are reversed and then the left & right are trimmed.

This means that --left actually trims from the right of the read in the SAM/BAM for reverse reads.

Optionally --ignoreStrand/-i can be specified to ignore the strand information and treat forward/reverse the same.

trimBam will modify the sequences to 'N', and the quality string to '!' unless the optional parameter --clip/-c is specified. If --clip/-c is specified, the ends will be soft clipped instead of modified.

== Soft Clipping Notes (--clip/-c) ==
Available in version 1.0.14 and later.

When soft clipping:
:* if the entire read would be soft clipped, no clipping is done, and instead the read is marked as unmapped
:* mate information is not updated (start positions/mapping may change after soft clipping)
:** run samtools fixmate to fix mate information (will first need to sort by read name)
:* output is not sorted (start positions/mapping may change after soft clipping)
:** run samtools sort to resort by coordinate (after fixmate)
:* soft clips already in the read are maintained or added to
:** if 3 bases were clipped and 2 are specified to be clipped, no change is made to that end
:** if 3 bases were clipped and 5 are specified to be clipped, 2 additional bases are clipped from that end

=== Fixing the mate/resorting ===
In order to update the mate, samtools fixmate must be run.

In order to reorder the file, samtools sort must be run.

Notes about the samtools programs:
* samtools fixmate requires the file to be sorted by query name.
* samtools sort cannot write to pipes.

====Steps====
# Run this program and pipe it into samtools sort by query name
#* <pre>./bam trimBam <your InputFile> - [#basesToTrim] [any other options] -c | samtools sort -n - tempQuerySort</pre>
# Run samtools fixmate and pipe it into samtools sort by position
#* <pre> samtools fixmate tempQuerySort.bam - | samtools sort - finalResult</pre>

= Parameters =
<pre>
Required Parameters:
inFile : the SAM/BAM file to be read
outFile : the SAM/BAM file to be written
num-bases-to-trim-on-each-side : the number of bases/qualities to trim from each side
Instead of num-bases-to-trim-on-each-side, -L/-R (or --left/--right) can be specified to indicate the number of bases to trim from the left/right (left/right are reversed for reverse strands)
Optional Parameters:
--ignoreStrand : ignore strand information - do not reverse left/right for reverse reads
--clip : soft clip the ends rather than setting to N/!
</pre>
{{PhoneHomeParamDesc}}

== Required Parameters==

{{InBAMInputFile|noParam=1st}}
{{OutBAMOutputFile|noParam=2nd}}

==Optional parameters==
=== Number of Bases to Trim from Each End (3rd argument) ===
If the 3rd argument a number (with no flag/option), it is the number of bases to trim from each end of the reads.

===Trim Bases from the Left (<code>--left</code> or <code>-L</code>)===
Use <code>--left</code> or <code>-L</code> followed by the number of bases to be trimmed from the left.

By default reverse strands are reversed and then the left is trimmed, meaning that <code>--left</code> actually trims from the right of the read in the SAM/BAM for reverse reads.

Use [[#Ignore the Strand when Trimming (--ignoreStrand or -i)|--ignoreStrand/-i]] to ignore the strand information and treat forward/reverse the same.

===Trim Bases from the Right (<code>--right</code> or <code>-R</code>)===
Use <code>--right</code> or <code>-R</code> followed by the number of bases to be trimmed from the right.

By default reverse strands are reversed and then the right is trimmed, meaning that <code>--right</code> actually trims from the left of the read in the SAM/BAM for reverse reads.

Use [[#Ignore the Strand when Trimming (--ignoreStrand or -i)|--ignoreStrand/-i]] to ignore the strand information and treat forward/reverse the same.

=== Ignore the Strand when Trimming (<code>--ignoreStrand</code> or <code>-i</code>) ===
Use <code>--ignoreStrand</code> or <code>-i</code> to ignore the strand information and treat forward/reverse the same. When <code>--ignoreStrand</code> or <code>-i</code> is set, do not reverse reverse reads prior to trimming left/right.

=== SoftClip the Ends (<code>--clip</code> or <code>-c</code>) ===
Use <code>--clip</code> or <code>-c</code> to soft clip the ends instead of setting to N/! (or set to unmapped if the entire read would be soft clipped).

See [[#Soft Clipping Notes (--clip/-c)|Soft Clipping Notes]] for more information about clipping and post processing that will need to be done.

{{noeofBGZFParameter}}

{{PhoneHomeParameters}}

= Return Value =
Returns the SamStatus for the reads/writes. 0 on success, non-0 on failure.

= Examples =

==Trim the same number of bases from each side==
Example Input, trimming 2 bases:
<pre>
./bin trimBam testFiles/testSam.sam results/trimSam.sam 2
</pre>
Example Output:
<pre>
Arguments in effect:
Input file : testFiles/testSam.sam
Output file : results/trimSam.sam
#Bases to trim from each side : 2

Number of records read = 10
Number of records written = 10
</pre>

==Trim different bases from each side, but treat reverse strands the opposite==
Example Input, trimming 1 base from the left and 2 bases from the right for forward strands and do the opposite for reverse strands:
<pre>
./bin trimBam testFiles/testSam.sam results/trimSam.sam -L 1 -R 2
</pre>
Example Output:
<pre>
Arguments in effect:
Input file : testFiles/testSam.sam
Output file : results/trimSam.sam
#Bases to trim from the left of forward strands : 1
#Bases to trim from the right of forward strands: 2
#Bases to trim from the left of reverse strands : 2
#Bases to trim from the right of reverse strands : 1

Number of records read = 10
Number of records written = 10
</pre>

==Trim different bases from each side, but treat forward & reverse the same==
Example Input, trimming 1 base from the left and 2 bases from the right ignoring strand information:
<pre>
./bin trimBam testFiles/testSam.sam results/trimSam.sam -L 1 -R 2 --ignoreStrand
</pre>
Example Output:
<pre>
Arguments in effect:
Input file : testFiles/testSam.sam
Output file : results/trimSam.sam
#Bases to trim from the left of forward strands : 1
#Bases to trim from the right of forward strands: 2
#Bases to trim from the left of reverse strands : 1
#Bases to trim from the right of reverse strands : 2

Number of records read = 10
Number of records written = 10
</pre>

[[Category:BamUtil|trimBam]]
[[Category:BAM Software]]
[[Category:Software]]

BamUtil: trimBam

2017-11-13T03:28:02Z

Mktrost: /* Soft Clipping Notes (--clip/-c) */

= Overview of the <code>trimBam</code> function of <code>bamUtil</code> =
The <code>trimBam</code> option on the [[bamUtil]] executable trims the end of reads in a SAM/BAM file, changing read ends to ‘N’ and quality to ‘!’, or by soft clipping (if command-line option, <code>--clip</code> is specified).

= Usage =
./bam trimBam [inFile] [outFile] [num-bases-to-trim-on-each-side]

Version 1.0.6 and later:

Alternately, the number of bases from each side can be specified (either or both -L/-R (--left/--right) can be specified):
./bam trimBam [inFile] [outFile] -L [num-bases-to-trim-from-left] -R [num-bases-to-trim-from-right]

By default reverse strands are reversed and then the left & right are trimmed.

This means that --left actually trims from the right of the read in the SAM/BAM for reverse reads.

Optionally --ignoreStrand/-i can be specified to ignore the strand information and treat forward/reverse the same.

trimBam will modify the sequences to 'N', and the quality string to '!' unless the optional parameter --clip/-c is specified. If --clip/-c is specified, the ends will be soft clipped instead of modified.

== Soft Clipping Notes (--clip/-c) ==
(Version 1.0.14 and later)
When soft clipping:
:* if the entire read would be soft clipped, no clipping is done, and instead the read is marked as unmapped
:* mate information is not updated (start positions/mapping may change after soft clipping)
:** run samtools fixmate to fix mate information (will first need to sort by read name)
:* output is not sorted (start positions/mapping may change after soft clipping)
:** run samtools sort to resort by coordinate (after fixmate)
:* soft clips already in the read are maintained or added to
:** if 3 bases were clipped and 2 are specified to be clipped, no change is made to that end
:** if 3 bases were clipped and 5 are specified to be clipped, 2 additional bases are clipped from that end

=== Fixing the mate/resorting ===
In order to update the mate, samtools fixmate must be run.

In order to reorder the file, samtools sort must be run.

Notes about the samtools programs:
* samtools fixmate requires the file to be sorted by query name.
* samtools sort cannot write to pipes.

====Steps====
# Run this program and pipe it into samtools sort by query name
#* <pre>./bam trimBam <your InputFile> - [#basesToTrim] [any other options] -c | samtools sort -n - tempQuerySort</pre>
# Run samtools fixmate and pipe it into samtools sort by position
#* <pre> samtools fixmate tempQuerySort.bam - | samtools sort - finalResult</pre>

= Parameters =
<pre>
Required Parameters:
inFile : the SAM/BAM file to be read
outFile : the SAM/BAM file to be written
num-bases-to-trim-on-each-side : the number of bases/qualities to trim from each side
Instead of num-bases-to-trim-on-each-side, -L/-R (or --left/--right) can be specified to indicate the number of bases to trim from the left/right (left/right are reversed for reverse strands)
Optional Parameters:
--ignoreStrand : ignore strand information - do not reverse left/right for reverse reads
--clip : soft clip the ends rather than setting to N/!
</pre>
{{PhoneHomeParamDesc}}

== Required Parameters==

{{InBAMInputFile|noParam=1st}}
{{OutBAMOutputFile|noParam=2nd}}

==Optional parameters==
=== Number of Bases to Trim from Each End (3rd argument) ===
If the 3rd argument a number (with no flag/option), it is the number of bases to trim from each end of the reads.

===Trim Bases from the Left (<code>--left</code> or <code>-L</code>)===
Use <code>--left</code> or <code>-L</code> followed by the number of bases to be trimmed from the left.

By default reverse strands are reversed and then the left is trimmed, meaning that <code>--left</code> actually trims from the right of the read in the SAM/BAM for reverse reads.

Use [[#Ignore the Strand when Trimming (--ignoreStrand or -i)|--ignoreStrand/-i]] to ignore the strand information and treat forward/reverse the same.

===Trim Bases from the Right (<code>--right</code> or <code>-R</code>)===
Use <code>--right</code> or <code>-R</code> followed by the number of bases to be trimmed from the right.

By default reverse strands are reversed and then the right is trimmed, meaning that <code>--right</code> actually trims from the left of the read in the SAM/BAM for reverse reads.

Use [[#Ignore the Strand when Trimming (--ignoreStrand or -i)|--ignoreStrand/-i]] to ignore the strand information and treat forward/reverse the same.

=== Ignore the Strand when Trimming (<code>--ignoreStrand</code> or <code>-i</code>) ===
Use <code>--ignoreStrand</code> or <code>-i</code> to ignore the strand information and treat forward/reverse the same. When <code>--ignoreStrand</code> or <code>-i</code> is set, do not reverse reverse reads prior to trimming left/right.

=== SoftClip the Ends (<code>--clip</code> or <code>-c</code>) ===
Use <code>--clip</code> or <code>-c</code> to soft clip the ends instead of setting to N/! (or set to unmapped if the entire read would be soft clipped).

See [[#Soft Clipping Notes (--clip/-c)|Soft Clipping Notes]] for more information about clipping and post processing that will need to be done.

{{noeofBGZFParameter}}

{{PhoneHomeParameters}}

= Return Value =
Returns the SamStatus for the reads/writes. 0 on success, non-0 on failure.

= Examples =

==Trim the same number of bases from each side==
Example Input, trimming 2 bases:
<pre>
./bin trimBam testFiles/testSam.sam results/trimSam.sam 2
</pre>
Example Output:
<pre>
Arguments in effect:
Input file : testFiles/testSam.sam
Output file : results/trimSam.sam
#Bases to trim from each side : 2

Number of records read = 10
Number of records written = 10
</pre>

==Trim different bases from each side, but treat reverse strands the opposite==
Example Input, trimming 1 base from the left and 2 bases from the right for forward strands and do the opposite for reverse strands:
<pre>
./bin trimBam testFiles/testSam.sam results/trimSam.sam -L 1 -R 2
</pre>
Example Output:
<pre>
Arguments in effect:
Input file : testFiles/testSam.sam
Output file : results/trimSam.sam
#Bases to trim from the left of forward strands : 1
#Bases to trim from the right of forward strands: 2
#Bases to trim from the left of reverse strands : 2
#Bases to trim from the right of reverse strands : 1

Number of records read = 10
Number of records written = 10
</pre>

==Trim different bases from each side, but treat forward & reverse the same==
Example Input, trimming 1 base from the left and 2 bases from the right ignoring strand information:
<pre>
./bin trimBam testFiles/testSam.sam results/trimSam.sam -L 1 -R 2 --ignoreStrand
</pre>
Example Output:
<pre>
Arguments in effect:
Input file : testFiles/testSam.sam
Output file : results/trimSam.sam
#Bases to trim from the left of forward strands : 1
#Bases to trim from the right of forward strands: 2
#Bases to trim from the left of reverse strands : 1
#Bases to trim from the right of reverse strands : 2

Number of records read = 10
Number of records written = 10
</pre>

[[Category:BamUtil|trimBam]]
[[Category:BAM Software]]
[[Category:Software]]

VerifyBamID

2017-09-09T19:29:12Z

Mktrost: /* Command Line Options */

[[Category:Software|VerifyBamID]]

'''verifyBamID''' is a software that verifies whether the reads in particular file match previously known genotypes for an individual (or group of individuals), and checks whether the reads are contaminated as a mixture of two samples. '''verifyBamID''' can detect sample contamination and swaps when external genotypes are available. When external genotypes are not available, '''verifyBamID''' still robustly detects sample swaps.

== Download verifyBamID ==

To get a copy of verifyBamId, go to: https://github.com/statgen/verifyBamID/releases

Select the latest release and download in one of 3 ways:
# Binary expected to run in Ubuntu x64 platform. In other platforms, please download the source distribution and build it.
#* verifyBamID.#.#.#.gz
#* You will need to run "gunzip" on the .gz file
# Souce Code including libStatGen (uses a fixed version of libStatGen)
#* verifyBamIDLibStatGen.#.#.#.tgz
#* Run "tar xvf" on this file. Cd into the resulting directory & type make.
# Source Code without libStatGen (allows alternative/newer versions of libStatGen)
#* Source code (tar.gz) or Source code (zip)
#* You will need to download libStatGen separately if you do not already have it.

To get a copy of older releases go to the [http://csg.sph.umich.edu//kang/verifyBamID/download VerifyBamID Download] download page.

== Join in verifyBamID mailing list ==

Please join in the [http://groups.google.com/group/verifybamid VerifyBamID Google Group] to ask / discuss / comment about verifyBamID.

== What's new ==

(2014/02/13)
* Put verifyBamID in github.
* Added PhoneHome/Version Checking to VerifyBamID

(2012/06/20)
* Fixed a bug of incorrect estimate of contamination when --chip-full option was used (Thanks to Richard Smith)
* Fixed a bug of incorrect per-readgroup output in --chip-* parameter

(2012/05/24)
* Fixed a bug of incorrect per-readgroup output (Thanks to Matthew Flickinger)
* '''(IMPORTANT)''' Add an option to remove either side of overlapping fragment. This option is turned on by default, and can be turned off usig --ignoreOverlapPair. If your sequence data has very short insert size, this update may increase the sensitivity of estimated contamination.
* Changes in the directory structure and Makefile

(2012/05/18) The new release of verifyBamID have undergone major change since the last version (as of 2011 April). Here are the highlights
* The genotype / allele frequency file is now based on VCF format rather than PLINK format.
* The reference sequence information is no longer required
* Uses Brent's method for precise estimation of contamination parameters
* Generate the depth distribution statistics.
* Estimated reference-bias parameters (useful mostly for ABI SOLiD sequence data)

== Build verifyBamID ==

The binary download of verifyBamID is available. You may use that version in Ubuntu 64-bit platform.

If you download the source that includes libStatGen:
tar xvf verifyBamIDLibStatGen.#.#.#.tgz
cd verifyBamID_#.#.#
make
Executable: verifyBamID/bin/verifyBamID

If you download the source without libStatGen:
tar xvf verifyBamID-#.#.#.tar.gz
cd verifyBamID-1.1.0
make cloneLib (if ../libStatGen does not exist)
make
Executable: ./bin/verifyBamID

Note that '''make cloneLib''' command will create a directory ../libStatGen under your verifyBamID directory, and '''make''' will create binary of verifyBamID under verifyBamID/bin/

If you have a different version of libStatGen at that path, then skip the cloneLib step. If the libStatGen you want to use is at a different location then update verifyBamID's Makefile.inc. Replace: LIB_PATH_VERIFY_BAM_ID ?= $(LIB_PATH_GENERAL) with
LIB_PATH_VERIFY_BAM_ID = /path/to/libStatGen

verifyBamID is designed to be reasonably portable.

However, since development occurs only on Ubuntu (9.10-13.10) x86 and x64 platforms, and later, there are likely other portability issues.

== Basic Usage ==

A key step in any genetic analysis is to verify whether data being generated matches expectations. ''verifyBamID'' checks whether reads in a BAM file match previous genotypes for a specific sample. In addition, it detects possible sample mixture from population allele frequency only, which can be particularly useful when the genotype data is not available.

Using a mathematical model that relates observed sequence reads to an hypothetical true genotype, ''verifyBamID'' tries to decide whether sequence reads match a particular individual or are more likely to be contaminated (including a small proportion of foreign DNA), derived from a closely related individual, or derived from a completely different individual.

== Basic Usage Example ==

Here is a typical command line:

verifyBamID --vcf [input.vcf] --bam [input.bam] --out [output.prefix] --verbose --ignoreRG

where
[input.bam] is a BAM (Binary Alignment Map) file of a sequence reads
[input.vcf] is input VCF file containing individual genotypes or AF or AC/AN fields in the INFO field. gzipped VCF is also allowed.
[outPrefix] is output prefix of output files - [outPrefix].{selfRG,selfSM,bestRG,bestSM,depthRG,depthSM} will be created.

More detailed description of command line input is below

== Preparing input files ==

verifyBamID requires two input files - VCF file containing external genotypes or allele frequency information, and the BAM file.

=== VCF input genotype file ===

The input VCF file contains (1) external genotype information and/or (2) allele frequency information as AF entry or AC/AN entries in the INFO field. (See [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 | VCF specification] for further details). If neither information is provided, verifyBamID will not work properly.

If external genotype information is provided, sequence+array method will identify contamination and sample swaps by comparing the concordance between the external genotypes and the sequence reads. Additionally, sequence-only method will provide additional contamination estimates by modeling the sequence reads as mixture of two unknown samples based on the allele frequency information in the VCF file.

Input VCF file needs to meet several additional contraints need to meet in order to properly run verifyBamID.
* The VCF is assumed to be well-formed. For example, verifyBamID does not check whether REF allele actually matches with reference sequence.
* The VCF should only contain SNPs. Current version of verifyBamID does not accept INDELs, MNPs, Structural Variations, or other complex variants.
* The individual IDs in the VCF file, must be identical with the individual identifier in the BAM file. Otherwise, --smID option can override the sample ID information of the BAM file to the ID that matches to the individual IDs in the VCF file.
* IMPORTANT : For targeted sequencing data, it is important to subselect the markers to only include on-target markers in the genotype file. Off-target markers are not likely to have multiple non-duplicated reads at the marker position, and it may create artifacts in the analysis due to overlapping fragments.
* Currently, verifyBamID takes only autosomal chromosomes as input VCF.

An example input VCF file (without external genotype) is provided below. Note that AC and AC entries exists in the INFO field for the allele frequency information.

#CHROM POS ID REF ALT QUAL FILTER INFO
20 61651 SNP20-9651 C A . PASS CR=99.86851;GentrainScore=0.7055;HW=0.077647716;AN=2180;AC=11
20 63231 SNP20-11231 T G . PASS CR=99.93036;GentrainScore=0.7837;HW=0.035481825;AN=2182;AC=275
20 63244 rs6139074 A C . PASS CR=98.893394;GentrainScore=0.8001;HW=7.327299E-7;AN=2162;AC=501
20 63799 rs1418258 C T . PASS CR=99.75217;GentrainScore=0.8170;HW=0.6653377;AN=2182;AC=881

=== Input BAM file ===

verifyBamID requires a sorted, indexed, base quality recalibrated, and duplication-marked BAM file. It also requires to contain "@RG" header lines to annotation different readGroups (sequencing runs and lanes). The SM tag in the "@RG" header should match with one of the genotyped sample. Otherwise, verifyBamID may not be able to test whether the sequenced sample matches with genotyped sample, but will try to detect sample mixture from allele frequency, and will try to detect the best-matching sample among the genotyped sample.

== What the default option does ==

The default option of '''verifyBamID''' is the recommended setting for the most sequencing studies to provide a rapid and informative response. The default option provides the following features:
* --free-mix is turned on for estimating contamination using sequence-only method
* --chip-mix is turned on for estimating contamination or swap using sequence+array method, if the external genotype file is provided in the VCF
* --self is turnd on : The default option does not try to compare the sequence reads to identify the best matching individual (which is possible with --best option). It only compares with the external genotypes from the same individual to the sequenced individual.
* --maxDepth 20 is used without --precise option : The default option is intended for whole genome low coverage sequencing. For the targeted exome sequencing, --maxDepth 1000 and --precise is recommended.
* --ignoreRG is not a default option, but a recommended option, when you want to check the contamination for the entire BAM rather than examining each read group separately. This option will increase the computational efficiency especially in the case whether the sequence reads are multiplexed across many sequencing runs.

== Interpreting output files ==

See also [[Understanding VerifyBamID output]].

=== Output files ===
When verifyBamID runs successfully, the following sets of files may be generated.
* [outPrefix].selfSM - Per-sample statistics describing how well the sample matches to the annotated sample.
* [outPrefix].depthSM - The depth distribution of the sequence reads per sample
* [outPrefix].selfRG - Per-readGroup statistics describing how well each lane matches to the annotated sample. (available only without --ignoreRG option)
* [outPrefix].depthRG - The depth distribution of the sequence reads per readGroup. (available only without --ignoreRG option)
* [outPrefix].bestSM - Per-sample best-match statistics with best-matching sample among the genotyped sample (available only with --best option)
* [outPrefix].bestRG - Per-readgroup best-match statistics with best-matching sample among the genotyped sample (available only with --best and without --ignoreRG option)

=== Column information in the output files ===
The .selfSM/.selfRG/.bestSM/.bestRG files have the following 19 columns per sample, or per readgroup (lane).

# SEQ_SM : Sample ID of the sequenced sample. Obtained from @RG header / SM tag in the BAM file
# RG : ReadGroup ID of sequenced lane. For [outPrefix].selfSM and [outPrefix].bestSM, these values are "ALL"
# CHIP_ID : Sample ID compared to in the genotype file. For [outPrefix].selfRG and [outPrefix].selfSM, these values should be identical to [SEQ_SM] or "NA" if the genotype of sequenced samples are unavailable. For [outPrefix].bestRG and [outPrefix].bestSM, these values should be the ID of best-matching sample among the genotype files compared to.
# # SNPs : # of SNPs passing the criteria from the VCF file
# # READS : Total # of reads loaded from the BAM file
# # AVG_DP : Average sequencing depth at the sites in the VCF file
# FREEMIX : Sequence-only estimate of contamination (0-1 scale)
# FREELK1 : Maximum log-likelihood of the sequence reads given estimated contamination under sequence-only method
# FREELK0 : Log-likelihood of the sequence reads given no contamination under sequence-only method
# FREE_RH : Estimated reference bias parameter Pr(refBase|HET) (when --free-refBias or --free-full is used)
# FREE_RA : Estimated reference bias parameter Pr(refBase|HOMALT) (when --free-refBias or --free-full is used)
# CHIPMIX : Sequence+array estimate of contamination (NA if the external genotype is unavailable) (0-1 scale)
# CHIPLK1 : Maximum log-likelihood of the sequence reads given estimated contamination under sequence+array method (NA if the external genotypes are unavailable)
# CHIPLK0 : Log-likelihood of the sequence reads given no contamination under sequence+array method (NA if the external genotypes are unavailable)
# CHIP_RH : Estimated reference bias parameter Pr(refBase|HET) (when --chip-refBias or --chip-full is used)
# CHIP_RA : Estimated reference bias parameter Pr(refBase|HOMALT) (when --chip-refBias or --chip-full is used)
# DPREF : Depth (Coverage) of HomRef site (based on the genotypes of (SELF_SM/BEST_SM), passing mapQ, baseQual, maxDepth thresholds.
# RDPHET : DPHET/DPREF, Relative depth to HomRef site at Heterozygous site.
# RDPALT : DPHET/DPREF, Relative depth to HomRef site at HomAlt site.

=== A guideline to interpret output files ===

verifyBamID provides a series of information that is informative to determine whether the sample is possibly contaminated or swapped, but there is no single criteria that works for every circumstances. There are a few unmodeled factor in the estimation of [SELF-IBD]/[BEST-IBD] and [%MIX], so please note that the MLE estimation may not always exactly match to the true amount of contamination. Here we provide a guideline to flag potentially contaminated/swapped samples

* Each sample or lane can be checked in this way. When [CHIPMIX] >> 0.02 and/or [FREEMIX] >> 0.02, meaning 2% or more of non-reference bases are observed in reference sites, we recommend to examine the data more carefully for the possibility of contamination.
* We recommend to check each lane for the possibility of sample swaps. When [CHIPMIX] ~ 1 AND [FREEMIX] ~ 0, then it is possible that the sample is swapped with another sample. When [CHIPMIX] ~ 0 in .bestSM file, [CHIP_ID] might be actually the swapped sample. Otherwise, the swapped sample may not exist in the genotype data you have compared.
* When genotype data is not available but allele-frequency-based estimates of [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, then it is possible that the sample is contaminated with other sample. We recommend to use per-sample data rather than per-lane data for checking this for low coverage data, because the inference will be more confident when there are large number of bases with depth 2 or higher.

== Command Line Options ==

The following parameters are available. Ones with "[]" are in effect:

Available Options
Input Files : --vcf [], --bam [], --subset [],
--smID []
VCF analysis options : --genoError [1.0e-03],
--minAF [0.01],
--minCallRate [0.50]
Individuals to compare with chip data : --site, --self, --best
Chip-free optimization options : --free-none, --free-mix [ON],
--free-refBias, --free-full
With-chip optimization options : --chip-none, --chip-mix [ON],
--chip-refBias, --chip-full
BAM analysis options : --ignoreRG, --ignoreOverlapPair,
--noEOF, --precise, --minMapQ [10],
--maxDepth [20], --minQ [13],
--maxQ [40], --grid [0.05]
Modeling Reference Bias : --refRef [1.00], --refHet [0.50],
--refAlt [0.00]
Output options : --out [], --verbose
PhoneHome : --noPhoneHome,
--phoneHomeThinning [50]

Each option provides the following features:
* --vcf : specify required VCF file
* --bam : specify required BAM file (indexed with .bam.bai or .bai file)
* --subset : list of individual IDs to calculate the allele frequency. All individuals are used if unspecified
* --smID : If the individual ID in the BAM file and VCF file does not match, substitute the BAM file's ID into the specified argument
* --genoError : error rate of the external genotype file
* --minAF : minimum allele frequency of the markers to include
* --minAF : minimum call rate of the markers to include
* --site : If set, use only site information in the VCF and do not compare with the actual genotypes
* --self : Only compare the ID-matching individuals between the VCF and BAM file
* --best : Find the best matching individuals (.bestSM and .bestRG files will be produced). This option is substantially longer than the default option
* --free-none : Do not perform sequence-only method to estimate parameters
* --free-mix : (default) Estimate contamination using sequence-only method with Brent's single dimensional optimization.
* --free-refBias : Estimate the reference bias parameters using sequence-only method with Simplex method
* --free-full : Estimate both reference bias parameters and the contamination parameters using sequence-only method
* --chip-none : Do not perform sequence+array method to estimate parameters
* --chip-mix : (default) Estimate contamination using sequence+array method with Brent's single dimensional optimization.
* --chip-refBias : Estimate the refernece bias parameters using sequence+array method with Simplex method
* --chip-full : Estimate both reference bias parameters and the contamination parameters using sequence+array method
* --ignoreRG : ignore the read grouup level comparison and compare samples only (recommended for an expedited run)
* --ignoreOverlapPair : ignore overlapping pair end fragment covering the same base. Disabling this option may decrease the sensitivity of the method when the insert size is short (with slight gain in the computational speed)
* --noEOF : do not check the EOF marker of the BAM file (for earlier version of BAM)
* --precise : calculate the likelihood in log-scale for high-depth data (recommended when --maxDepth is greater than 20. Can be a little bit slower)
* --minMapQ : minimum mapping quality of the sequence reads to compare
* --minQ : minimum base quality to include
* --maxQ : maximum base quality to cap
* --grid : the grid interval to search the optimum before running Brent's algorithm.
* --refRef : Initial Pr(refBase|HOMREFGeno) parameter
* --refHet : Initial Pr(refBase|HETGeno) parameter
* --refAlt : Initial Pr(refBase|HOMALTGeno) parameter
* --out : output file prefix (required)
* --verbose : print the progress of the method on the screeen
{{PhoneHomeParameters|hdr=====|bullet=1}}

== Principle of Operation ==

Each read group in a BAM file is evaluated independently. This means that in file with multiple read groups, problems will be flagged at the read group level (a plus). However, it also means that it might be hard to discern the correct assignment of read groups with very little data.

For each aligned base that overlaps a known genotype, we calculate the probability the probability that it was derived from a particular known genotype. This comparison considers only bases that overlap previously known genotypes and that meet the base quality and mapping quality thresholds.

Each individual in a pedigree has a different combination of genotypes, and bamGenotypeCheck will systematically search for the individual whose genotypes best match the observed read data.

For more about the technical details, see the page [[Verifying Sample Identities - Implementation]]

== Reference ==

Please cite the following paper:

G. Jun, M. Flickinger, K. N. Hetrick, Kurt, J. M. Romm, K. F. Doheny, G. Abecasis, M. Boehnke,and H. M. Kang, ''Detecting and Estimating Contamination of Human DNA Samples in Sequencing and Array-Based Genotype Data'', American journal of human genetics doi:10.1016/j.ajhg.2012.09.004 (volume 91 issue 5 pp.839 - 848)

== Contamination in Array Data ==

[[VerifyIDintensity]] or [[BAFRegress]] can estimate sample contamination from Illumina genotype array data.

== Acknowledgements ==

VerifyBamID is a result from collaborative effort by Hyun Min Kang, Goo Jun, Matthew Flickinger, Mary Kate Wing, Goncalo Abecasis, and Michael Boehnke. Please email to Hyun Min Kang [hmkang@umich.edu ] for any questions.

BamUtil: stats

2017-08-24T19:59:05Z

Mktrost: /* BaseQC (--pBaseQC and --cBaseQC and --baseSum) */

= Overview of the <code>stats</code> function of <code>bamUtil</code> =

The <code>stats</code> option on the [[BamUtil]] executable generates the specified statistics on a SAM/BAM file.

== Troubleshooting ==
See [[BamUtil:_FAQ#BamUtil:_stats|BamUtil: FAQ -> BamUtil: stats]] for troubleshooting help.

= Usage =
<pre>
./bam stats --in <inputFile> [--basic] [--qual] [--phred] [--pBaseQC <outputFileName>] [--cBaseQC <outputFileName>] [--maxNumReads <maxNum>][--unmapped] [--bamIndex <bamIndexFile>] [--regionList <regFileName>] [--requiredFlags <integerRequiredFlags>] [--excludeFlags <integerExcludeFlags>] [--noeof] [--params] [--withinRegion] [--baseSum] [--bufferSize <buffSize>] [--minMapQual <minMapQ>] [--dbsnp <dbsnpFile>]
</pre>

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to calculate stats for
Types of Statistics that can be generated:
--basic : Turn on basic statistic generation
--qual : Generate a count for each quality (displayed as non-phred quality)
--phred : Generate a count for each quality (displayed as phred quality)
--pBaseQC : Write per base statistics as Percentages to the specified file. (use - for stdout)
pBaseQC & cBaseQC cannot both be specified.
--cBaseQC : Write per base statistics as Counts to the specified file. (use - for stdout)
pBaseQC & cBaseQC cannot both be specified.
Optional Parameters:
--maxNumReads : Maximum number of reads to process
Defaults to -1 to indicate all reads.
--unmapped : Only process unmapped reads (requires a bamIndex file)
--bamIndex : The path/name of the bam index file
(if required and not specified, uses the --in value + ".bai")
--regionList : File containing the regions to be processed chr<tab>start_pos<tab>end_pos.
Positions are 0 based and the end_pos is not included in the region.
Uses bamIndex.
--excludeFlags : Skip any records with any of the specified flags set
(specify an integer representation of the flags)
--requiredFlags : Only process records with all of the specified flags set
(specify an integer representation of the flags)
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings.
Optional phred/qual Only Parameters:
--withinRegion : Only count qualities if they fall within regions specified.
Only applicable if regionList is also specified.
Optional BaseQC Only Parameters:
--baseSum : Print an overall summary of the baseQC for the file to stderr.
--bufferSize : Size of the pileup buffer for calculating the BaseQC parameters.
Default: 1024
--minMapQual : The minimum mapping quality for filtering reads in the baseQC stats.
--dbsnp : The dbSnp file of positions to exclude from baseQC analysis.
</pre>
{{PhoneHomeParamDesc}}

== Required Parameters ==

{{inBAMInputFile}}

== Optional Parameters ==
=== Maximum number of reads to process(<code>--maxNumReads</code>) ===
Use <code>--maxNumReads</code> followed by a number to indicate the maximum number of reads to process before exiting. By default, it is set to -1 to indicate all reads should be processed.

=== Only Process Unmapped Reads (<code>--unmapped</code>) ===
Use <code>--unmapped</code> to process only unmapped reads.

This parameter requires [[#Bam Index File (--bamIndex)|<code>--bamIndex</code>]].

{{BamIndex}}

=== Only Process Certain Regions (<code>--regionList</code>) ===
Use <code>--regionList</code> followed by the filename to process only the regions specified in the file.

The positions in the file are specified one per line with the following format: <nowiki>chr<tab>start_pos<tab>end_pos.</nowiki>

Positions are 0 based and the end_pos is not included in the region.

This parameter requires [[#Bam Index File (--bamIndex)|<code>--bamIndex</code>]].

=== Exclude Flags (<code>--excludeFlags</code>) ===
Use <code>--excludeFlags</code> followed by an integer representation of the flags to only process reads with any of the specified flags set.

=== Required Flags (<code>--requiredFlags</code>) ===
Use <code>--requiredFlags</code> followed by an integer representation of the flags to only process records with all of the specified flags set.

== Types of Statistics ==

=== Basic (<code>--basic</code>) ===

Prints summary statistics for the file:

*TotalReads - # of reads that are in the file
*MappedReads - # of reads marked mapped in the flag
*PairedReads - # of reads marked paired in the flag
*ProperPair - # of reads marked paired AND proper paired in the flag
*DuplicateReads - # of reads marked duplicate in the flag
*QCFailureReads - # of reads marked QC failure in the flag
*MappingRate(%) - # of reads marked mapped in the flag / TotalReads
*PairedReads(%) - # of reads marked paired in the flag / TotalReads
*ProperPair(%) - # of reads marked paired AND proper paired in the flag / TotalReads
*DupRate(%) - # of reads marked duplicate in the flag / TotalReads
*QCFailRate(%) - # of reads marked QC failure in the flag / TotalReads
*TotalBases - # of bases in all reads
*BasesInMappedReads - # of bases in reads marked mapped in the flag

=== Qual/Phred (<code>--phred</code> and <code>--qual</code>) ===

Prints a count of the number of times each quality value appears in the file to stderr.

*<code>phred</code> Displays Quality as phred integers [0-93]
*<code>qual</code> Displays Quality as non-phred integers (phred + 33) [33-126]

By default, these counts include all qualities in the BAM file.

To exclude unmapped reads and soft clips, use --excludeFlags 4.

To only include records that overlap a set of regions, use --regionList and specify a bed file with the regions. If a read overlaps the region, all qualities will be counted even if those bases do not fall in the region. If you only want to count qualities that fall within the region, also specify --withinRegion. Without excluding unmapped reads, it will include soft clips that overlap the region.

==== Optional Phred/Qual Only Parameters ====
===== Within Region (<code>--withinRegion</code>) =====
Use <code>--withinRegion</code> with [[#Qual/Phred (--phred and --qual)|<code>--phred</code> or <code>--qual</code>]] options to only count qualities if they fall within the regions specified using [[#Only Process Certain Regions (--regionList)|<code>--regionList</code>]] (only applicable if [[#Only Process Certain Regions (--regionList)|<code>--regionList</code>]] is also specified).

=== BaseQC (<code>--pBaseQC</code> and <code>--cBaseQC</code> and <code>--baseSum</code>) ===

The <code>pBaseQC</code> and <code>cBaseQC</code> options generate per base statistics. Only one of these two options can be specified. They write statistics generated for each position to the file specified after the option (use <code>-</code> to write to STDOUT). They use the same logic for calculating statistics, but <code>pBaseQC</code> writes the statistics as percentages, and <code>cBaseQC</code> writes them as counts. The order of the statistics are also different.

The <code>baseSum</code> option can be used with either <code>pBaseQC</code> or <code>cBaseQC</code> or on its own. <code>baseSum</code> generates a summary of the per position statistics and writes it to stderr. It calculates the per position base statistics even if they will not be written anywhere (neither <code>pBaseQC</code> nor <code>cBaseQC</code> are specified).

All three options use the same logic for calculating the statistics:
* A read spans a position if the read starts at or before the position, ends at or after the position and the position is not a clip. CIGAR operations allowed for the position are M/X/=/D/N. If the CIGAR is '*', only numbers for the specified reference position are incremented.
*Currently there is no special logic to exclude positions/reads where the reference base is 'N' or the read base is 'N'.

 

==== Percentage-Based Output Format (<code>--pBaseQC</code>) ====

Order/Descriptions:

{|border=1
! Field !! Description !!style="width: 80px"| Excludes Duplicates, QC Failures !!style="width: 80px"| Excludes Unmapped !!style="width: 80px"| Excludes MapQual = 255 !!style="width: 80px"| Excludes Below Min MapQual !!style="width: 80px"| Excludes CIGAR Deletions, Skips
|-
| chrom || Chromosome/reference name string from the SAM/BAM
|-
| chromStart || 0-based start position
|-
| chromEnd || 0-based end position (always 1 greater than start and not included in this region)
|-
| Depth || # of reads that are mapped with acceptable Mapping Quality, and are not duplicates or QC failures || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20Bases || # of bases at this position with a base quality (from the read) of Q20 or higher || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20BasesPct(%) || Q20Bases / Depth || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| TotalReads || # of reads that span this position || || || || ||
|-
| MappedBases || # of reads marked mapped in the flag || align="center"|X || align="center"|X || || ||
|-
| MappingRate(%) || MappedBases / TotalReads || align="center"|X || align="center"|X || || ||
|-
| MapRate_MQPass(%) || # of reads that have a Mapping Quality >= a minimum Mapping Quality / TotalReads || align="center"|X || align="center"|X || || ||
|-
| ZeroMapQual(%) || # of reads that have a Mapping Quality of 0 / TotalReads || align="center"|X || align="center"|X || || ||
|-
| MapQual<10(%) || # of reads that have a Mapping Quality < 10 / TotalReads || align="center"|X || align="center"|X || || ||
|-
| PairedReads(%) || # of reads marked paired in the flag / TotalReads || align="center"|X || align="center"|X || || ||
|-
| ProperPaired(%) || # of reads marked paired AND proper paired in the flag / TotalReads || align="center"|X || align="center"|X || || ||
|-
| DupRate(%) || # of reads marked duplicate in the flag / TotalReads || || || || ||
|-
| QCFailRate(%) || # of reads marked QC failure in the flag / TotalReads || || || || ||
|-
| AverageMapQuality || sum of included mapping qualities / AverageMapQualCount || align="center"|X || align="center"|X || align="center"|X || ||
|-
| AverageMapQualCount || # of mapping qualities in AverageMapQuality || align="center"|X || align="center"|X || align="center"|X || ||
|-
|}

This output does not include a MapQual255 count.

===== Sample Output =====
<pre>chrom chromStart chromEnd Depth Q20Bases Q20BasesPct(%) TotalReads MappedBases MappingRate(%) MapRate_MQPass(%) ZeroMapQual(%) MapQual<10(%) PairedReads(%) ProperPaired(%) DupRate(%) QCFailRate(%) AverageMapQuality AverageMapQualCount
1 100 101 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 101 102 2 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 102 103 0 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 0.000 0
1 103 104 0 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 0.000 0
1 104 105 2 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 105 106 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 110 111 0 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 0.000 0
1 111 112 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 112 113 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 10012 10013 14 0 0.000 42 33 78.571 52.381 26.190 52.381 85.714 35.714 14.286 14.286 11.000 21
1 10013 10014 14 10 71.429 39 30 76.923 51.282 25.641 51.282 84.615 38.462 15.385 15.385 11.000 21
1 10023 10024 0 0 0.000 39 30 76.923 51.282 25.641 51.282 84.615 38.462 15.385 15.385 0.000 0
1 10024 10025 14 12 85.714 39 30 76.923 51.282 25.641 51.282 84.615 38.462 15.385 15.385 11.000 21
</pre>

==== Count-Based Output Format (<code>--cBaseQC</code>) ====
Order/Descriptions:
{|border=1
! Field !! Description !!style="width: 80px"| Excludes Duplicates, QC Failures !!style="width: 80px"| Excludes Unmapped !!style="width: 80px"| Excludes MapQual = 255 !!style="width: 80px"| Excludes Below Min MapQual !!style="width: 80px"| Excludes CIGAR Deletions, Skips
|-
| chrom || Chromosome/reference name string from the SAM/BAM
|-
| chromStart || 0-based start position
|-
| chromEnd || 0-based end position (always 1 greater than start and not included in this region)
|-
| TotalReads || # of reads that span this position || || || || ||
|-
| Dups || # of reads marked duplicate in the flag || || || || ||
|-
| QCFail || # of reads marked QC failure in the flag || || || || ||
|-
| Mapped || # of reads marked mapped in the flag || align="center"|X || align="center"|X || || ||
|-
| Paired || # of reads marked paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ProperPaired || # of reads marked paired AND proper paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ZeroMapQual || # of reads that have a Mapping Quality of 0 || align="center"|X || align="center"|X || || ||
|-
| MapQual<10(%) || # of reads that have a Mapping Quality < 10 || align="center"|X || align="center"|X || || ||
|-
| MapQual255 || # of reads that have a Mapping Quality = 255 || align="center"|X || align="center"|X || || ||
|-
| PassMapQual || # of reads that have a Mapping Quality >= a minimum Mapping Quality || align="center"|X || align="center"|X || || ||
|-
| AverageMapQuality || sum of included mapping qualities / AverageMapQualCount || align="center"|X || align="center"|X || align="center"|X || ||
|-
| AverageMapQualCount || # of mapping qualities in AverageMapQuality || align="center"|X || align="center"|X || align="center"|X ||
|- ||
| Depth || # of reads that are mapped with acceptable Mapping Quality, and are not duplicates or QC failures || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20Bases || # of bases at this position with a base quality (from the read) of Q20 or higher || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
|}

==== Summary of per Position Statistics (<code>--baseSum</code>) ====
Use <code>--baseSum</code> to print an overall summary of the baseQC for the file to stderr.

This option can be used with or without <code>--pBaseQC</code> and <code>--cBaseQC</code>

The values are tab delimited. First there is a header line describing the summary. The next line has the Means, and the last line has the Standard Deviations.

{|border=1
! Field !! Description !!style="width: 80px"| Excludes Duplicates, QC Failures !!style="width: 80px"| Excludes Unmapped !!style="width: 80px"| Excludes MapQual = 255 !!style="width: 80px"| Excludes Below Min MapQual !!style="width: 80px"| Excludes CIGAR Deletions, Skips
|-
| TotalReads || # of reads that span this position || || || || ||
|-
| Dups || # of reads marked duplicate in the flag || || || || ||
|-
| QCFail || # of reads marked QC failure in the flag || || || || ||
|-
| Mapped || # of reads marked mapped in the flag || align="center"|X || align="center"|X || || ||
|-
| Paired || # of reads marked paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ProperPaired || # of reads marked paired AND proper paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ZeroMapQual || # of reads that have a Mapping Quality of 0 || align="center"|X || align="center"|X || || ||
|-
| MapQual<10(%) || # of reads that have a Mapping Quality < 10 || align="center"|X || align="center"|X || || ||
|-
| MapQual255 || # of reads that have a Mapping Quality = 255 || align="center"|X || align="center"|X || || ||
|-
| PassMapQual || # of reads that have a Mapping Quality >= a minimum Mapping Quality || align="center"|X || align="center"|X || || ||
|-
| AverageMapQuality || sum of included mapping qualities / AverageMapQualCount || align="center"|X || align="center"|X || align="center"|X || ||
|-
| AverageMapQualCount || # of mapping qualities in AverageMapQuality || align="center"|X || align="center"|X || align="center"|X ||
|- ||
| Depth || # of reads that are mapped with acceptable Mapping Quality, and are not duplicates or QC failures || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20Bases || # of bases at this position with a base quality (from the read) of Q20 or higher || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
|}

===== Sample Output =====
<pre>
Summary of Pileup Stats (1st Row is Mean, 2nd Row is Standard Deviation)
TotalReads Dups QCFail Mapped Paired ProperPaired ZeroMapQual MapQual<10 MapQual255 PassMapQual AverageMapQuality AverageMapQualCount
Depth Q20Bases
14.307692 1.846154 1.846154 8.769231 7.846154 0.923077 2.923077 5.846154 0.000000 2.923077 11.000000 8.769231 2.076923 1.153846
17.670053 2.882307 2.882307 9.038380 7.603137 1.441153 3.012793 6.025586 0.000000 3.012793 0.000000 9.038380 2.841993 1.993579
</pre>

==== Optional BaseQC Only Parameters ====
===== Pileup Buffer Size (<code>--bufferSize</code>) =====
Use the <code>--bufferSize</code> option followed by the size of the pileup buffer to use for [[BaseQC (--pBaseQC and --cBaseQC and --baseSum)|baseQC]] stats.

===== Minimum Mapping Quality (<code>--minMapQual</code>) =====
Use the <code>--minMapQual</code> option followed by the minimum mapping quality for filtering reads in the [[BaseQC (--pBaseQC and --cBaseQC and --baseSum)|baseQC]] stats.

===== DBSNP File (<code>--dbsnp</code>) =====
Use the <code>--dbsnp</code> option followed by the name of the dbsnp file to specify the positions to exclude from [[BaseQC (--pBaseQC and --cBaseQC and --baseSum)|baseQC]] analysis.

{{PhoneHomeParameters}}

= Return Value =
0 on Success, non-0 on failure

[[Category:BamUtil|stats]] [[Category:BAM_Software]] [[Category:Software]]

BamUtil: stats

2017-08-24T19:58:21Z

Mktrost: /* BaseQC (--pBaseQC and --cBaseQC and --baseSum) */

= Overview of the <code>stats</code> function of <code>bamUtil</code> =

The <code>stats</code> option on the [[BamUtil]] executable generates the specified statistics on a SAM/BAM file.

== Troubleshooting ==
See [[BamUtil:_FAQ#BamUtil:_stats|BamUtil: FAQ -> BamUtil: stats]] for troubleshooting help.

= Usage =
<pre>
./bam stats --in <inputFile> [--basic] [--qual] [--phred] [--pBaseQC <outputFileName>] [--cBaseQC <outputFileName>] [--maxNumReads <maxNum>][--unmapped] [--bamIndex <bamIndexFile>] [--regionList <regFileName>] [--requiredFlags <integerRequiredFlags>] [--excludeFlags <integerExcludeFlags>] [--noeof] [--params] [--withinRegion] [--baseSum] [--bufferSize <buffSize>] [--minMapQual <minMapQ>] [--dbsnp <dbsnpFile>]
</pre>

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to calculate stats for
Types of Statistics that can be generated:
--basic : Turn on basic statistic generation
--qual : Generate a count for each quality (displayed as non-phred quality)
--phred : Generate a count for each quality (displayed as phred quality)
--pBaseQC : Write per base statistics as Percentages to the specified file. (use - for stdout)
pBaseQC & cBaseQC cannot both be specified.
--cBaseQC : Write per base statistics as Counts to the specified file. (use - for stdout)
pBaseQC & cBaseQC cannot both be specified.
Optional Parameters:
--maxNumReads : Maximum number of reads to process
Defaults to -1 to indicate all reads.
--unmapped : Only process unmapped reads (requires a bamIndex file)
--bamIndex : The path/name of the bam index file
(if required and not specified, uses the --in value + ".bai")
--regionList : File containing the regions to be processed chr<tab>start_pos<tab>end_pos.
Positions are 0 based and the end_pos is not included in the region.
Uses bamIndex.
--excludeFlags : Skip any records with any of the specified flags set
(specify an integer representation of the flags)
--requiredFlags : Only process records with all of the specified flags set
(specify an integer representation of the flags)
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings.
Optional phred/qual Only Parameters:
--withinRegion : Only count qualities if they fall within regions specified.
Only applicable if regionList is also specified.
Optional BaseQC Only Parameters:
--baseSum : Print an overall summary of the baseQC for the file to stderr.
--bufferSize : Size of the pileup buffer for calculating the BaseQC parameters.
Default: 1024
--minMapQual : The minimum mapping quality for filtering reads in the baseQC stats.
--dbsnp : The dbSnp file of positions to exclude from baseQC analysis.
</pre>
{{PhoneHomeParamDesc}}

== Required Parameters ==

{{inBAMInputFile}}

== Optional Parameters ==
=== Maximum number of reads to process(<code>--maxNumReads</code>) ===
Use <code>--maxNumReads</code> followed by a number to indicate the maximum number of reads to process before exiting. By default, it is set to -1 to indicate all reads should be processed.

=== Only Process Unmapped Reads (<code>--unmapped</code>) ===
Use <code>--unmapped</code> to process only unmapped reads.

This parameter requires [[#Bam Index File (--bamIndex)|<code>--bamIndex</code>]].

{{BamIndex}}

=== Only Process Certain Regions (<code>--regionList</code>) ===
Use <code>--regionList</code> followed by the filename to process only the regions specified in the file.

The positions in the file are specified one per line with the following format: <nowiki>chr<tab>start_pos<tab>end_pos.</nowiki>

Positions are 0 based and the end_pos is not included in the region.

This parameter requires [[#Bam Index File (--bamIndex)|<code>--bamIndex</code>]].

=== Exclude Flags (<code>--excludeFlags</code>) ===
Use <code>--excludeFlags</code> followed by an integer representation of the flags to only process reads with any of the specified flags set.

=== Required Flags (<code>--requiredFlags</code>) ===
Use <code>--requiredFlags</code> followed by an integer representation of the flags to only process records with all of the specified flags set.

== Types of Statistics ==

=== Basic (<code>--basic</code>) ===

Prints summary statistics for the file:

*TotalReads - # of reads that are in the file
*MappedReads - # of reads marked mapped in the flag
*PairedReads - # of reads marked paired in the flag
*ProperPair - # of reads marked paired AND proper paired in the flag
*DuplicateReads - # of reads marked duplicate in the flag
*QCFailureReads - # of reads marked QC failure in the flag
*MappingRate(%) - # of reads marked mapped in the flag / TotalReads
*PairedReads(%) - # of reads marked paired in the flag / TotalReads
*ProperPair(%) - # of reads marked paired AND proper paired in the flag / TotalReads
*DupRate(%) - # of reads marked duplicate in the flag / TotalReads
*QCFailRate(%) - # of reads marked QC failure in the flag / TotalReads
*TotalBases - # of bases in all reads
*BasesInMappedReads - # of bases in reads marked mapped in the flag

=== Qual/Phred (<code>--phred</code> and <code>--qual</code>) ===

Prints a count of the number of times each quality value appears in the file to stderr.

*<code>phred</code> Displays Quality as phred integers [0-93]
*<code>qual</code> Displays Quality as non-phred integers (phred + 33) [33-126]

By default, these counts include all qualities in the BAM file.

To exclude unmapped reads and soft clips, use --excludeFlags 4.

To only include records that overlap a set of regions, use --regionList and specify a bed file with the regions. If a read overlaps the region, all qualities will be counted even if those bases do not fall in the region. If you only want to count qualities that fall within the region, also specify --withinRegion. Without excluding unmapped reads, it will include soft clips that overlap the region.

==== Optional Phred/Qual Only Parameters ====
===== Within Region (<code>--withinRegion</code>) =====
Use <code>--withinRegion</code> with [[#Qual/Phred (--phred and --qual)|<code>--phred</code> or <code>--qual</code>]] options to only count qualities if they fall within the regions specified using [[#Only Process Certain Regions (--regionList)|<code>--regionList</code>]] (only applicable if [[#Only Process Certain Regions (--regionList)|<code>--regionList</code>]] is also specified).

=== BaseQC (<code>--pBaseQC</code> and <code>--cBaseQC</code> and <code>--baseSum</code>) ===

The <code>pBaseQC</code> and <code>cBaseQC</code> options generate per base statistics. Only one of these two options can be specified. They write statistics generated for each position to the file specified after the option (use '-' to write to STDOUT). They use the same logic for calculating statistics, but <code>pBaseQC</code> writes the statistics as percentages, and <code>cBaseQC</code> writes them as counts. The order of the statistics are also different.

The <code>baseSum</code> option can be used with either <code>pBaseQC</code> or <code>cBaseQC</code> or on its own. <code>baseSum</code> generates a summary of the per position statistics and writes it to stderr. It calculates the per position base statistics even if they will not be written anywhere (neither <code>pBaseQC</code> nor <code>cBaseQC</code> are specified).

All three options use the same logic for calculating the statistics:
* A read spans a position if the read starts at or before the position, ends at or after the position and the position is not a clip. CIGAR operations allowed for the position are M/X/=/D/N. If the CIGAR is '*', only numbers for the specified reference position are incremented.
*Currently there is no special logic to exclude positions/reads where the reference base is 'N' or the read base is 'N'.

 

==== Percentage-Based Output Format (<code>--pBaseQC</code>) ====

Order/Descriptions:

{|border=1
! Field !! Description !!style="width: 80px"| Excludes Duplicates, QC Failures !!style="width: 80px"| Excludes Unmapped !!style="width: 80px"| Excludes MapQual = 255 !!style="width: 80px"| Excludes Below Min MapQual !!style="width: 80px"| Excludes CIGAR Deletions, Skips
|-
| chrom || Chromosome/reference name string from the SAM/BAM
|-
| chromStart || 0-based start position
|-
| chromEnd || 0-based end position (always 1 greater than start and not included in this region)
|-
| Depth || # of reads that are mapped with acceptable Mapping Quality, and are not duplicates or QC failures || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20Bases || # of bases at this position with a base quality (from the read) of Q20 or higher || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20BasesPct(%) || Q20Bases / Depth || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| TotalReads || # of reads that span this position || || || || ||
|-
| MappedBases || # of reads marked mapped in the flag || align="center"|X || align="center"|X || || ||
|-
| MappingRate(%) || MappedBases / TotalReads || align="center"|X || align="center"|X || || ||
|-
| MapRate_MQPass(%) || # of reads that have a Mapping Quality >= a minimum Mapping Quality / TotalReads || align="center"|X || align="center"|X || || ||
|-
| ZeroMapQual(%) || # of reads that have a Mapping Quality of 0 / TotalReads || align="center"|X || align="center"|X || || ||
|-
| MapQual<10(%) || # of reads that have a Mapping Quality < 10 / TotalReads || align="center"|X || align="center"|X || || ||
|-
| PairedReads(%) || # of reads marked paired in the flag / TotalReads || align="center"|X || align="center"|X || || ||
|-
| ProperPaired(%) || # of reads marked paired AND proper paired in the flag / TotalReads || align="center"|X || align="center"|X || || ||
|-
| DupRate(%) || # of reads marked duplicate in the flag / TotalReads || || || || ||
|-
| QCFailRate(%) || # of reads marked QC failure in the flag / TotalReads || || || || ||
|-
| AverageMapQuality || sum of included mapping qualities / AverageMapQualCount || align="center"|X || align="center"|X || align="center"|X || ||
|-
| AverageMapQualCount || # of mapping qualities in AverageMapQuality || align="center"|X || align="center"|X || align="center"|X || ||
|-
|}

This output does not include a MapQual255 count.

===== Sample Output =====
<pre>chrom chromStart chromEnd Depth Q20Bases Q20BasesPct(%) TotalReads MappedBases MappingRate(%) MapRate_MQPass(%) ZeroMapQual(%) MapQual<10(%) PairedReads(%) ProperPaired(%) DupRate(%) QCFailRate(%) AverageMapQuality AverageMapQualCount
1 100 101 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 101 102 2 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 102 103 0 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 0.000 0
1 103 104 0 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 0.000 0
1 104 105 2 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 105 106 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 110 111 0 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 0.000 0
1 111 112 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 112 113 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 10012 10013 14 0 0.000 42 33 78.571 52.381 26.190 52.381 85.714 35.714 14.286 14.286 11.000 21
1 10013 10014 14 10 71.429 39 30 76.923 51.282 25.641 51.282 84.615 38.462 15.385 15.385 11.000 21
1 10023 10024 0 0 0.000 39 30 76.923 51.282 25.641 51.282 84.615 38.462 15.385 15.385 0.000 0
1 10024 10025 14 12 85.714 39 30 76.923 51.282 25.641 51.282 84.615 38.462 15.385 15.385 11.000 21
</pre>

==== Count-Based Output Format (<code>--cBaseQC</code>) ====
Order/Descriptions:
{|border=1
! Field !! Description !!style="width: 80px"| Excludes Duplicates, QC Failures !!style="width: 80px"| Excludes Unmapped !!style="width: 80px"| Excludes MapQual = 255 !!style="width: 80px"| Excludes Below Min MapQual !!style="width: 80px"| Excludes CIGAR Deletions, Skips
|-
| chrom || Chromosome/reference name string from the SAM/BAM
|-
| chromStart || 0-based start position
|-
| chromEnd || 0-based end position (always 1 greater than start and not included in this region)
|-
| TotalReads || # of reads that span this position || || || || ||
|-
| Dups || # of reads marked duplicate in the flag || || || || ||
|-
| QCFail || # of reads marked QC failure in the flag || || || || ||
|-
| Mapped || # of reads marked mapped in the flag || align="center"|X || align="center"|X || || ||
|-
| Paired || # of reads marked paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ProperPaired || # of reads marked paired AND proper paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ZeroMapQual || # of reads that have a Mapping Quality of 0 || align="center"|X || align="center"|X || || ||
|-
| MapQual<10(%) || # of reads that have a Mapping Quality < 10 || align="center"|X || align="center"|X || || ||
|-
| MapQual255 || # of reads that have a Mapping Quality = 255 || align="center"|X || align="center"|X || || ||
|-
| PassMapQual || # of reads that have a Mapping Quality >= a minimum Mapping Quality || align="center"|X || align="center"|X || || ||
|-
| AverageMapQuality || sum of included mapping qualities / AverageMapQualCount || align="center"|X || align="center"|X || align="center"|X || ||
|-
| AverageMapQualCount || # of mapping qualities in AverageMapQuality || align="center"|X || align="center"|X || align="center"|X ||
|- ||
| Depth || # of reads that are mapped with acceptable Mapping Quality, and are not duplicates or QC failures || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20Bases || # of bases at this position with a base quality (from the read) of Q20 or higher || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
|}

==== Summary of per Position Statistics (<code>--baseSum</code>) ====
Use <code>--baseSum</code> to print an overall summary of the baseQC for the file to stderr.

This option can be used with or without <code>--pBaseQC</code> and <code>--cBaseQC</code>

The values are tab delimited. First there is a header line describing the summary. The next line has the Means, and the last line has the Standard Deviations.

{|border=1
! Field !! Description !!style="width: 80px"| Excludes Duplicates, QC Failures !!style="width: 80px"| Excludes Unmapped !!style="width: 80px"| Excludes MapQual = 255 !!style="width: 80px"| Excludes Below Min MapQual !!style="width: 80px"| Excludes CIGAR Deletions, Skips
|-
| TotalReads || # of reads that span this position || || || || ||
|-
| Dups || # of reads marked duplicate in the flag || || || || ||
|-
| QCFail || # of reads marked QC failure in the flag || || || || ||
|-
| Mapped || # of reads marked mapped in the flag || align="center"|X || align="center"|X || || ||
|-
| Paired || # of reads marked paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ProperPaired || # of reads marked paired AND proper paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ZeroMapQual || # of reads that have a Mapping Quality of 0 || align="center"|X || align="center"|X || || ||
|-
| MapQual<10(%) || # of reads that have a Mapping Quality < 10 || align="center"|X || align="center"|X || || ||
|-
| MapQual255 || # of reads that have a Mapping Quality = 255 || align="center"|X || align="center"|X || || ||
|-
| PassMapQual || # of reads that have a Mapping Quality >= a minimum Mapping Quality || align="center"|X || align="center"|X || || ||
|-
| AverageMapQuality || sum of included mapping qualities / AverageMapQualCount || align="center"|X || align="center"|X || align="center"|X || ||
|-
| AverageMapQualCount || # of mapping qualities in AverageMapQuality || align="center"|X || align="center"|X || align="center"|X ||
|- ||
| Depth || # of reads that are mapped with acceptable Mapping Quality, and are not duplicates or QC failures || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20Bases || # of bases at this position with a base quality (from the read) of Q20 or higher || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
|}

===== Sample Output =====
<pre>
Summary of Pileup Stats (1st Row is Mean, 2nd Row is Standard Deviation)
TotalReads Dups QCFail Mapped Paired ProperPaired ZeroMapQual MapQual<10 MapQual255 PassMapQual AverageMapQuality AverageMapQualCount
Depth Q20Bases
14.307692 1.846154 1.846154 8.769231 7.846154 0.923077 2.923077 5.846154 0.000000 2.923077 11.000000 8.769231 2.076923 1.153846
17.670053 2.882307 2.882307 9.038380 7.603137 1.441153 3.012793 6.025586 0.000000 3.012793 0.000000 9.038380 2.841993 1.993579
</pre>

==== Optional BaseQC Only Parameters ====
===== Pileup Buffer Size (<code>--bufferSize</code>) =====
Use the <code>--bufferSize</code> option followed by the size of the pileup buffer to use for [[BaseQC (--pBaseQC and --cBaseQC and --baseSum)|baseQC]] stats.

===== Minimum Mapping Quality (<code>--minMapQual</code>) =====
Use the <code>--minMapQual</code> option followed by the minimum mapping quality for filtering reads in the [[BaseQC (--pBaseQC and --cBaseQC and --baseSum)|baseQC]] stats.

===== DBSNP File (<code>--dbsnp</code>) =====
Use the <code>--dbsnp</code> option followed by the name of the dbsnp file to specify the positions to exclude from [[BaseQC (--pBaseQC and --cBaseQC and --baseSum)|baseQC]] analysis.

{{PhoneHomeParameters}}

= Return Value =
0 on Success, non-0 on failure

[[Category:BamUtil|stats]] [[Category:BAM_Software]] [[Category:Software]]

BamUtil: stats

2017-08-24T19:57:14Z

Mktrost: /* Parameters */

= Overview of the <code>stats</code> function of <code>bamUtil</code> =

The <code>stats</code> option on the [[BamUtil]] executable generates the specified statistics on a SAM/BAM file.

== Troubleshooting ==
See [[BamUtil:_FAQ#BamUtil:_stats|BamUtil: FAQ -> BamUtil: stats]] for troubleshooting help.

= Usage =
<pre>
./bam stats --in <inputFile> [--basic] [--qual] [--phred] [--pBaseQC <outputFileName>] [--cBaseQC <outputFileName>] [--maxNumReads <maxNum>][--unmapped] [--bamIndex <bamIndexFile>] [--regionList <regFileName>] [--requiredFlags <integerRequiredFlags>] [--excludeFlags <integerExcludeFlags>] [--noeof] [--params] [--withinRegion] [--baseSum] [--bufferSize <buffSize>] [--minMapQual <minMapQ>] [--dbsnp <dbsnpFile>]
</pre>

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to calculate stats for
Types of Statistics that can be generated:
--basic : Turn on basic statistic generation
--qual : Generate a count for each quality (displayed as non-phred quality)
--phred : Generate a count for each quality (displayed as phred quality)
--pBaseQC : Write per base statistics as Percentages to the specified file. (use - for stdout)
pBaseQC & cBaseQC cannot both be specified.
--cBaseQC : Write per base statistics as Counts to the specified file. (use - for stdout)
pBaseQC & cBaseQC cannot both be specified.
Optional Parameters:
--maxNumReads : Maximum number of reads to process
Defaults to -1 to indicate all reads.
--unmapped : Only process unmapped reads (requires a bamIndex file)
--bamIndex : The path/name of the bam index file
(if required and not specified, uses the --in value + ".bai")
--regionList : File containing the regions to be processed chr<tab>start_pos<tab>end_pos.
Positions are 0 based and the end_pos is not included in the region.
Uses bamIndex.
--excludeFlags : Skip any records with any of the specified flags set
(specify an integer representation of the flags)
--requiredFlags : Only process records with all of the specified flags set
(specify an integer representation of the flags)
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings.
Optional phred/qual Only Parameters:
--withinRegion : Only count qualities if they fall within regions specified.
Only applicable if regionList is also specified.
Optional BaseQC Only Parameters:
--baseSum : Print an overall summary of the baseQC for the file to stderr.
--bufferSize : Size of the pileup buffer for calculating the BaseQC parameters.
Default: 1024
--minMapQual : The minimum mapping quality for filtering reads in the baseQC stats.
--dbsnp : The dbSnp file of positions to exclude from baseQC analysis.
</pre>
{{PhoneHomeParamDesc}}

== Required Parameters ==

{{inBAMInputFile}}

== Optional Parameters ==
=== Maximum number of reads to process(<code>--maxNumReads</code>) ===
Use <code>--maxNumReads</code> followed by a number to indicate the maximum number of reads to process before exiting. By default, it is set to -1 to indicate all reads should be processed.

=== Only Process Unmapped Reads (<code>--unmapped</code>) ===
Use <code>--unmapped</code> to process only unmapped reads.

This parameter requires [[#Bam Index File (--bamIndex)|<code>--bamIndex</code>]].

{{BamIndex}}

=== Only Process Certain Regions (<code>--regionList</code>) ===
Use <code>--regionList</code> followed by the filename to process only the regions specified in the file.

The positions in the file are specified one per line with the following format: <nowiki>chr<tab>start_pos<tab>end_pos.</nowiki>

Positions are 0 based and the end_pos is not included in the region.

This parameter requires [[#Bam Index File (--bamIndex)|<code>--bamIndex</code>]].

=== Exclude Flags (<code>--excludeFlags</code>) ===
Use <code>--excludeFlags</code> followed by an integer representation of the flags to only process reads with any of the specified flags set.

=== Required Flags (<code>--requiredFlags</code>) ===
Use <code>--requiredFlags</code> followed by an integer representation of the flags to only process records with all of the specified flags set.

== Types of Statistics ==

=== Basic (<code>--basic</code>) ===

Prints summary statistics for the file:

*TotalReads - # of reads that are in the file
*MappedReads - # of reads marked mapped in the flag
*PairedReads - # of reads marked paired in the flag
*ProperPair - # of reads marked paired AND proper paired in the flag
*DuplicateReads - # of reads marked duplicate in the flag
*QCFailureReads - # of reads marked QC failure in the flag
*MappingRate(%) - # of reads marked mapped in the flag / TotalReads
*PairedReads(%) - # of reads marked paired in the flag / TotalReads
*ProperPair(%) - # of reads marked paired AND proper paired in the flag / TotalReads
*DupRate(%) - # of reads marked duplicate in the flag / TotalReads
*QCFailRate(%) - # of reads marked QC failure in the flag / TotalReads
*TotalBases - # of bases in all reads
*BasesInMappedReads - # of bases in reads marked mapped in the flag

=== Qual/Phred (<code>--phred</code> and <code>--qual</code>) ===

Prints a count of the number of times each quality value appears in the file to stderr.

*<code>phred</code> Displays Quality as phred integers [0-93]
*<code>qual</code> Displays Quality as non-phred integers (phred + 33) [33-126]

By default, these counts include all qualities in the BAM file.

To exclude unmapped reads and soft clips, use --excludeFlags 4.

To only include records that overlap a set of regions, use --regionList and specify a bed file with the regions. If a read overlaps the region, all qualities will be counted even if those bases do not fall in the region. If you only want to count qualities that fall within the region, also specify --withinRegion. Without excluding unmapped reads, it will include soft clips that overlap the region.

==== Optional Phred/Qual Only Parameters ====
===== Within Region (<code>--withinRegion</code>) =====
Use <code>--withinRegion</code> with [[#Qual/Phred (--phred and --qual)|<code>--phred</code> or <code>--qual</code>]] options to only count qualities if they fall within the regions specified using [[#Only Process Certain Regions (--regionList)|<code>--regionList</code>]] (only applicable if [[#Only Process Certain Regions (--regionList)|<code>--regionList</code>]] is also specified).

=== BaseQC (<code>--pBaseQC</code> and <code>--cBaseQC</code> and <code>--baseSum</code>) ===

The <code>pBaseQC</code> and <code>cBaseQC</code> options generate per base statistics. Only one of these two options can be specified. They write statistics generated for each position to the file specified after the option. They use the same logic for calculating statistics, but <code>pBaseQC</code> writes the statistics as percentages, and <code>cBaseQC</code> writes them as counts. The order of the statistics are also different.

The <code>baseSum</code> option can be used with either <code>pBaseQC</code> or <code>cBaseQC</code> or on its own. <code>baseSum</code> generates a summary of the per position statistics and writes it to stderr. It calculates the per position base statistics even if they will not be written anywhere (neither <code>pBaseQC</code> nor <code>cBaseQC</code> are specified).

All three options use the same logic for calculating the statistics:
* A read spans a position if the read starts at or before the position, ends at or after the position and the position is not a clip. CIGAR operations allowed for the position are M/X/=/D/N. If the CIGAR is '*', only numbers for the specified reference position are incremented.
*Currently there is no special logic to exclude positions/reads where the reference base is 'N' or the read base is 'N'.

 

==== Percentage-Based Output Format (<code>--pBaseQC</code>) ====

Order/Descriptions:

{|border=1
! Field !! Description !!style="width: 80px"| Excludes Duplicates, QC Failures !!style="width: 80px"| Excludes Unmapped !!style="width: 80px"| Excludes MapQual = 255 !!style="width: 80px"| Excludes Below Min MapQual !!style="width: 80px"| Excludes CIGAR Deletions, Skips
|-
| chrom || Chromosome/reference name string from the SAM/BAM
|-
| chromStart || 0-based start position
|-
| chromEnd || 0-based end position (always 1 greater than start and not included in this region)
|-
| Depth || # of reads that are mapped with acceptable Mapping Quality, and are not duplicates or QC failures || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20Bases || # of bases at this position with a base quality (from the read) of Q20 or higher || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20BasesPct(%) || Q20Bases / Depth || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| TotalReads || # of reads that span this position || || || || ||
|-
| MappedBases || # of reads marked mapped in the flag || align="center"|X || align="center"|X || || ||
|-
| MappingRate(%) || MappedBases / TotalReads || align="center"|X || align="center"|X || || ||
|-
| MapRate_MQPass(%) || # of reads that have a Mapping Quality >= a minimum Mapping Quality / TotalReads || align="center"|X || align="center"|X || || ||
|-
| ZeroMapQual(%) || # of reads that have a Mapping Quality of 0 / TotalReads || align="center"|X || align="center"|X || || ||
|-
| MapQual<10(%) || # of reads that have a Mapping Quality < 10 / TotalReads || align="center"|X || align="center"|X || || ||
|-
| PairedReads(%) || # of reads marked paired in the flag / TotalReads || align="center"|X || align="center"|X || || ||
|-
| ProperPaired(%) || # of reads marked paired AND proper paired in the flag / TotalReads || align="center"|X || align="center"|X || || ||
|-
| DupRate(%) || # of reads marked duplicate in the flag / TotalReads || || || || ||
|-
| QCFailRate(%) || # of reads marked QC failure in the flag / TotalReads || || || || ||
|-
| AverageMapQuality || sum of included mapping qualities / AverageMapQualCount || align="center"|X || align="center"|X || align="center"|X || ||
|-
| AverageMapQualCount || # of mapping qualities in AverageMapQuality || align="center"|X || align="center"|X || align="center"|X || ||
|-
|}

This output does not include a MapQual255 count.

===== Sample Output =====
<pre>chrom chromStart chromEnd Depth Q20Bases Q20BasesPct(%) TotalReads MappedBases MappingRate(%) MapRate_MQPass(%) ZeroMapQual(%) MapQual<10(%) PairedReads(%) ProperPaired(%) DupRate(%) QCFailRate(%) AverageMapQuality AverageMapQualCount
1 100 101 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 101 102 2 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 102 103 0 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 0.000 0
1 103 104 0 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 0.000 0
1 104 105 2 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 105 106 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 110 111 0 0 0.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 0.000 0
1 111 112 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 112 113 2 2 100.000 3 3 100.000 66.667 33.333 66.667 100.000 0.000 0.000 0.000 11.000 3
1 10012 10013 14 0 0.000 42 33 78.571 52.381 26.190 52.381 85.714 35.714 14.286 14.286 11.000 21
1 10013 10014 14 10 71.429 39 30 76.923 51.282 25.641 51.282 84.615 38.462 15.385 15.385 11.000 21
1 10023 10024 0 0 0.000 39 30 76.923 51.282 25.641 51.282 84.615 38.462 15.385 15.385 0.000 0
1 10024 10025 14 12 85.714 39 30 76.923 51.282 25.641 51.282 84.615 38.462 15.385 15.385 11.000 21
</pre>

==== Count-Based Output Format (<code>--cBaseQC</code>) ====
Order/Descriptions:
{|border=1
! Field !! Description !!style="width: 80px"| Excludes Duplicates, QC Failures !!style="width: 80px"| Excludes Unmapped !!style="width: 80px"| Excludes MapQual = 255 !!style="width: 80px"| Excludes Below Min MapQual !!style="width: 80px"| Excludes CIGAR Deletions, Skips
|-
| chrom || Chromosome/reference name string from the SAM/BAM
|-
| chromStart || 0-based start position
|-
| chromEnd || 0-based end position (always 1 greater than start and not included in this region)
|-
| TotalReads || # of reads that span this position || || || || ||
|-
| Dups || # of reads marked duplicate in the flag || || || || ||
|-
| QCFail || # of reads marked QC failure in the flag || || || || ||
|-
| Mapped || # of reads marked mapped in the flag || align="center"|X || align="center"|X || || ||
|-
| Paired || # of reads marked paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ProperPaired || # of reads marked paired AND proper paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ZeroMapQual || # of reads that have a Mapping Quality of 0 || align="center"|X || align="center"|X || || ||
|-
| MapQual<10(%) || # of reads that have a Mapping Quality < 10 || align="center"|X || align="center"|X || || ||
|-
| MapQual255 || # of reads that have a Mapping Quality = 255 || align="center"|X || align="center"|X || || ||
|-
| PassMapQual || # of reads that have a Mapping Quality >= a minimum Mapping Quality || align="center"|X || align="center"|X || || ||
|-
| AverageMapQuality || sum of included mapping qualities / AverageMapQualCount || align="center"|X || align="center"|X || align="center"|X || ||
|-
| AverageMapQualCount || # of mapping qualities in AverageMapQuality || align="center"|X || align="center"|X || align="center"|X ||
|- ||
| Depth || # of reads that are mapped with acceptable Mapping Quality, and are not duplicates or QC failures || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20Bases || # of bases at this position with a base quality (from the read) of Q20 or higher || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
|}

==== Summary of per Position Statistics (<code>--baseSum</code>) ====
Use <code>--baseSum</code> to print an overall summary of the baseQC for the file to stderr.

This option can be used with or without <code>--pBaseQC</code> and <code>--cBaseQC</code>

The values are tab delimited. First there is a header line describing the summary. The next line has the Means, and the last line has the Standard Deviations.

{|border=1
! Field !! Description !!style="width: 80px"| Excludes Duplicates, QC Failures !!style="width: 80px"| Excludes Unmapped !!style="width: 80px"| Excludes MapQual = 255 !!style="width: 80px"| Excludes Below Min MapQual !!style="width: 80px"| Excludes CIGAR Deletions, Skips
|-
| TotalReads || # of reads that span this position || || || || ||
|-
| Dups || # of reads marked duplicate in the flag || || || || ||
|-
| QCFail || # of reads marked QC failure in the flag || || || || ||
|-
| Mapped || # of reads marked mapped in the flag || align="center"|X || align="center"|X || || ||
|-
| Paired || # of reads marked paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ProperPaired || # of reads marked paired AND proper paired in the flag || align="center"|X || align="center"|X || || ||
|-
| ZeroMapQual || # of reads that have a Mapping Quality of 0 || align="center"|X || align="center"|X || || ||
|-
| MapQual<10(%) || # of reads that have a Mapping Quality < 10 || align="center"|X || align="center"|X || || ||
|-
| MapQual255 || # of reads that have a Mapping Quality = 255 || align="center"|X || align="center"|X || || ||
|-
| PassMapQual || # of reads that have a Mapping Quality >= a minimum Mapping Quality || align="center"|X || align="center"|X || || ||
|-
| AverageMapQuality || sum of included mapping qualities / AverageMapQualCount || align="center"|X || align="center"|X || align="center"|X || ||
|-
| AverageMapQualCount || # of mapping qualities in AverageMapQuality || align="center"|X || align="center"|X || align="center"|X ||
|- ||
| Depth || # of reads that are mapped with acceptable Mapping Quality, and are not duplicates or QC failures || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
| Q20Bases || # of bases at this position with a base quality (from the read) of Q20 or higher || align="center"|X || align="center"|X || align="center"|X || align="center"|X || align="center"|X
|-
|}

===== Sample Output =====
<pre>
Summary of Pileup Stats (1st Row is Mean, 2nd Row is Standard Deviation)
TotalReads Dups QCFail Mapped Paired ProperPaired ZeroMapQual MapQual<10 MapQual255 PassMapQual AverageMapQuality AverageMapQualCount
Depth Q20Bases
14.307692 1.846154 1.846154 8.769231 7.846154 0.923077 2.923077 5.846154 0.000000 2.923077 11.000000 8.769231 2.076923 1.153846
17.670053 2.882307 2.882307 9.038380 7.603137 1.441153 3.012793 6.025586 0.000000 3.012793 0.000000 9.038380 2.841993 1.993579
</pre>

==== Optional BaseQC Only Parameters ====
===== Pileup Buffer Size (<code>--bufferSize</code>) =====
Use the <code>--bufferSize</code> option followed by the size of the pileup buffer to use for [[BaseQC (--pBaseQC and --cBaseQC and --baseSum)|baseQC]] stats.

===== Minimum Mapping Quality (<code>--minMapQual</code>) =====
Use the <code>--minMapQual</code> option followed by the minimum mapping quality for filtering reads in the [[BaseQC (--pBaseQC and --cBaseQC and --baseSum)|baseQC]] stats.

===== DBSNP File (<code>--dbsnp</code>) =====
Use the <code>--dbsnp</code> option followed by the name of the dbsnp file to specify the positions to exclude from [[BaseQC (--pBaseQC and --cBaseQC and --baseSum)|baseQC]] analysis.

{{PhoneHomeParameters}}

= Return Value =
0 on Success, non-0 on failure

[[Category:BamUtil|stats]] [[Category:BAM_Software]] [[Category:Software]]

VerifyBamID

2017-06-14T08:20:37Z

Mktrost: /* Column information in the output files */

[[Category:Software|VerifyBamID]]

'''verifyBamID''' is a software that verifies whether the reads in particular file match previously known genotypes for an individual (or group of individuals), and checks whether the reads are contaminated as a mixture of two samples. '''verifyBamID''' can detect sample contamination and swaps when external genotypes are available. When external genotypes are not available, '''verifyBamID''' still robustly detects sample swaps.

== Download verifyBamID ==

To get a copy of verifyBamId, go to: https://github.com/statgen/verifyBamID/releases

Select the latest release and download in one of 3 ways:
# Binary expected to run in Ubuntu x64 platform. In other platforms, please download the source distribution and build it.
#* verifyBamID.#.#.#.gz
#* You will need to run "gunzip" on the .gz file
# Souce Code including libStatGen (uses a fixed version of libStatGen)
#* verifyBamIDLibStatGen.#.#.#.tgz
#* Run "tar xvf" on this file. Cd into the resulting directory & type make.
# Source Code without libStatGen (allows alternative/newer versions of libStatGen)
#* Source code (tar.gz) or Source code (zip)
#* You will need to download libStatGen separately if you do not already have it.

To get a copy of older releases go to the [http://csg.sph.umich.edu//kang/verifyBamID/download VerifyBamID Download] download page.

== Join in verifyBamID mailing list ==

Please join in the [http://groups.google.com/group/verifybamid VerifyBamID Google Group] to ask / discuss / comment about verifyBamID.

== What's new ==

(2014/02/13)
* Put verifyBamID in github.
* Added PhoneHome/Version Checking to VerifyBamID

(2012/06/20)
* Fixed a bug of incorrect estimate of contamination when --chip-full option was used (Thanks to Richard Smith)
* Fixed a bug of incorrect per-readgroup output in --chip-* parameter

(2012/05/24)
* Fixed a bug of incorrect per-readgroup output (Thanks to Matthew Flickinger)
* '''(IMPORTANT)''' Add an option to remove either side of overlapping fragment. This option is turned on by default, and can be turned off usig --ignoreOverlapPair. If your sequence data has very short insert size, this update may increase the sensitivity of estimated contamination.
* Changes in the directory structure and Makefile

(2012/05/18) The new release of verifyBamID have undergone major change since the last version (as of 2011 April). Here are the highlights
* The genotype / allele frequency file is now based on VCF format rather than PLINK format.
* The reference sequence information is no longer required
* Uses Brent's method for precise estimation of contamination parameters
* Generate the depth distribution statistics.
* Estimated reference-bias parameters (useful mostly for ABI SOLiD sequence data)

== Build verifyBamID ==

The binary download of verifyBamID is available. You may use that version in Ubuntu 64-bit platform.

If you download the source that includes libStatGen:
tar xvf verifyBamIDLibStatGen.#.#.#.tgz
cd verifyBamID_#.#.#
make
Executable: verifyBamID/bin/verifyBamID

If you download the source without libStatGen:
tar xvf verifyBamID-#.#.#.tar.gz
cd verifyBamID-1.1.0
make cloneLib (if ../libStatGen does not exist)
make
Executable: ./bin/verifyBamID

Note that '''make cloneLib''' command will create a directory ../libStatGen under your verifyBamID directory, and '''make''' will create binary of verifyBamID under verifyBamID/bin/

If you have a different version of libStatGen at that path, then skip the cloneLib step. If the libStatGen you want to use is at a different location then update verifyBamID's Makefile.inc. Replace: LIB_PATH_VERIFY_BAM_ID ?= $(LIB_PATH_GENERAL) with
LIB_PATH_VERIFY_BAM_ID = /path/to/libStatGen

verifyBamID is designed to be reasonably portable.

However, since development occurs only on Ubuntu (9.10-13.10) x86 and x64 platforms, and later, there are likely other portability issues.

== Basic Usage ==

A key step in any genetic analysis is to verify whether data being generated matches expectations. ''verifyBamID'' checks whether reads in a BAM file match previous genotypes for a specific sample. In addition, it detects possible sample mixture from population allele frequency only, which can be particularly useful when the genotype data is not available.

Using a mathematical model that relates observed sequence reads to an hypothetical true genotype, ''verifyBamID'' tries to decide whether sequence reads match a particular individual or are more likely to be contaminated (including a small proportion of foreign DNA), derived from a closely related individual, or derived from a completely different individual.

== Basic Usage Example ==

Here is a typical command line:

verifyBamID --vcf [input.vcf] --bam [input.bam] --out [output.prefix] --verbose --ignoreRG

where
[input.bam] is a BAM (Binary Alignment Map) file of a sequence reads
[input.vcf] is input VCF file containing individual genotypes or AF or AC/AN fields in the INFO field. gzipped VCF is also allowed.
[outPrefix] is output prefix of output files - [outPrefix].{selfRG,selfSM,bestRG,bestSM,depthRG,depthSM} will be created.

More detailed description of command line input is below

== Preparing input files ==

verifyBamID requires two input files - VCF file containing external genotypes or allele frequency information, and the BAM file.

=== VCF input genotype file ===

The input VCF file contains (1) external genotype information and/or (2) allele frequency information as AF entry or AC/AN entries in the INFO field. (See [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 | VCF specification] for further details). If neither information is provided, verifyBamID will not work properly.

If external genotype information is provided, sequence+array method will identify contamination and sample swaps by comparing the concordance between the external genotypes and the sequence reads. Additionally, sequence-only method will provide additional contamination estimates by modeling the sequence reads as mixture of two unknown samples based on the allele frequency information in the VCF file.

Input VCF file needs to meet several additional contraints need to meet in order to properly run verifyBamID.
* The VCF is assumed to be well-formed. For example, verifyBamID does not check whether REF allele actually matches with reference sequence.
* The VCF should only contain SNPs. Current version of verifyBamID does not accept INDELs, MNPs, Structural Variations, or other complex variants.
* The individual IDs in the VCF file, must be identical with the individual identifier in the BAM file. Otherwise, --smID option can override the sample ID information of the BAM file to the ID that matches to the individual IDs in the VCF file.
* IMPORTANT : For targeted sequencing data, it is important to subselect the markers to only include on-target markers in the genotype file. Off-target markers are not likely to have multiple non-duplicated reads at the marker position, and it may create artifacts in the analysis due to overlapping fragments.
* Currently, verifyBamID takes only autosomal chromosomes as input VCF.

An example input VCF file (without external genotype) is provided below. Note that AC and AC entries exists in the INFO field for the allele frequency information.

#CHROM POS ID REF ALT QUAL FILTER INFO
20 61651 SNP20-9651 C A . PASS CR=99.86851;GentrainScore=0.7055;HW=0.077647716;AN=2180;AC=11
20 63231 SNP20-11231 T G . PASS CR=99.93036;GentrainScore=0.7837;HW=0.035481825;AN=2182;AC=275
20 63244 rs6139074 A C . PASS CR=98.893394;GentrainScore=0.8001;HW=7.327299E-7;AN=2162;AC=501
20 63799 rs1418258 C T . PASS CR=99.75217;GentrainScore=0.8170;HW=0.6653377;AN=2182;AC=881

=== Input BAM file ===

verifyBamID requires a sorted, indexed, base quality recalibrated, and duplication-marked BAM file. It also requires to contain "@RG" header lines to annotation different readGroups (sequencing runs and lanes). The SM tag in the "@RG" header should match with one of the genotyped sample. Otherwise, verifyBamID may not be able to test whether the sequenced sample matches with genotyped sample, but will try to detect sample mixture from allele frequency, and will try to detect the best-matching sample among the genotyped sample.

== What the default option does ==

The default option of '''verifyBamID''' is the recommended setting for the most sequencing studies to provide a rapid and informative response. The default option provides the following features:
* --free-mix is turned on for estimating contamination using sequence-only method
* --chip-mix is turned on for estimating contamination or swap using sequence+array method, if the external genotype file is provided in the VCF
* --self is turnd on : The default option does not try to compare the sequence reads to identify the best matching individual (which is possible with --best option). It only compares with the external genotypes from the same individual to the sequenced individual.
* --maxDepth 20 is used without --precise option : The default option is intended for whole genome low coverage sequencing. For the targeted exome sequencing, --maxDepth 1000 and --precise is recommended.
* --ignoreRG is not a default option, but a recommended option, when you want to check the contamination for the entire BAM rather than examining each read group separately. This option will increase the computational efficiency especially in the case whether the sequence reads are multiplexed across many sequencing runs.

== Interpreting output files ==

See also [[Understanding VerifyBamID output]].

=== Output files ===
When verifyBamID runs successfully, the following sets of files may be generated.
* [outPrefix].selfSM - Per-sample statistics describing how well the sample matches to the annotated sample.
* [outPrefix].depthSM - The depth distribution of the sequence reads per sample
* [outPrefix].selfRG - Per-readGroup statistics describing how well each lane matches to the annotated sample. (available only without --ignoreRG option)
* [outPrefix].depthRG - The depth distribution of the sequence reads per readGroup. (available only without --ignoreRG option)
* [outPrefix].bestSM - Per-sample best-match statistics with best-matching sample among the genotyped sample (available only with --best option)
* [outPrefix].bestRG - Per-readgroup best-match statistics with best-matching sample among the genotyped sample (available only with --best and without --ignoreRG option)

=== Column information in the output files ===
The .selfSM/.selfRG/.bestSM/.bestRG files have the following 19 columns per sample, or per readgroup (lane).

# SEQ_SM : Sample ID of the sequenced sample. Obtained from @RG header / SM tag in the BAM file
# RG : ReadGroup ID of sequenced lane. For [outPrefix].selfSM and [outPrefix].bestSM, these values are "ALL"
# CHIP_ID : Sample ID compared to in the genotype file. For [outPrefix].selfRG and [outPrefix].selfSM, these values should be identical to [SEQ_SM] or "NA" if the genotype of sequenced samples are unavailable. For [outPrefix].bestRG and [outPrefix].bestSM, these values should be the ID of best-matching sample among the genotype files compared to.
# # SNPs : # of SNPs passing the criteria from the VCF file
# # READS : Total # of reads loaded from the BAM file
# # AVG_DP : Average sequencing depth at the sites in the VCF file
# FREEMIX : Sequence-only estimate of contamination (0-1 scale)
# FREELK1 : Maximum log-likelihood of the sequence reads given estimated contamination under sequence-only method
# FREELK0 : Log-likelihood of the sequence reads given no contamination under sequence-only method
# FREE_RH : Estimated reference bias parameter Pr(refBase|HET) (when --free-refBias or --free-full is used)
# FREE_RA : Estimated reference bias parameter Pr(refBase|HOMALT) (when --free-refBias or --free-full is used)
# CHIPMIX : Sequence+array estimate of contamination (NA if the external genotype is unavailable) (0-1 scale)
# CHIPLK1 : Maximum log-likelihood of the sequence reads given estimated contamination under sequence+array method (NA if the external genotypes are unavailable)
# CHIPLK0 : Log-likelihood of the sequence reads given no contamination under sequence+array method (NA if the external genotypes are unavailable)
# CHIP_RH : Estimated reference bias parameter Pr(refBase|HET) (when --chip-refBias or --chip-full is used)
# CHIP_RA : Estimated reference bias parameter Pr(refBase|HOMALT) (when --chip-refBias or --chip-full is used)
# DPREF : Depth (Coverage) of HomRef site (based on the genotypes of (SELF_SM/BEST_SM), passing mapQ, baseQual, maxDepth thresholds.
# RDPHET : DPHET/DPREF, Relative depth to HomRef site at Heterozygous site.
# RDPALT : DPHET/DPREF, Relative depth to HomRef site at HomAlt site.

=== A guideline to interpret output files ===

verifyBamID provides a series of information that is informative to determine whether the sample is possibly contaminated or swapped, but there is no single criteria that works for every circumstances. There are a few unmodeled factor in the estimation of [SELF-IBD]/[BEST-IBD] and [%MIX], so please note that the MLE estimation may not always exactly match to the true amount of contamination. Here we provide a guideline to flag potentially contaminated/swapped samples

* Each sample or lane can be checked in this way. When [CHIPMIX] >> 0.02 and/or [FREEMIX] >> 0.02, meaning 2% or more of non-reference bases are observed in reference sites, we recommend to examine the data more carefully for the possibility of contamination.
* We recommend to check each lane for the possibility of sample swaps. When [CHIPMIX] ~ 1 AND [FREEMIX] ~ 0, then it is possible that the sample is swapped with another sample. When [CHIPMIX] ~ 0 in .bestSM file, [CHIP_ID] might be actually the swapped sample. Otherwise, the swapped sample may not exist in the genotype data you have compared.
* When genotype data is not available but allele-frequency-based estimates of [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, then it is possible that the sample is contaminated with other sample. We recommend to use per-sample data rather than per-lane data for checking this for low coverage data, because the inference will be more confident when there are large number of bases with depth 2 or higher.

== Command Line Options ==

The following parameters are available. Ones with "[]" are in effect:

Available Options
Input Files : --vcf [], --bam [], --subset [],
--smID []
VCF analysis options : --genoError [1.0e-03],
--minAF [0.01],
--minCallRate [0.50]
Individuals to compare with chip data : --site, --self, --best
Chip-free optimization options : --free-none, --free-mix [ON],
--free-refBias, --free-full
With-chip optimization options : --chip-none, --chip-mix [ON],
--chip-refBias, --chip-full
BAM analysis options : --ignoreRG, --ignoreOverlapPair,
--noEOF, --precise, --minMapQ [10],
--maxDepth [20], --minQ [13],
--maxQ [40], --grid [0.05]
Modeling Reference Bias : --refRef [1.00], --refHet [0.50],
--refAlt [0.00]
Output options : --out [], --verbose
PhoneHome : --noPhoneHome,
--phoneHomeThinning [50]

Each option provides the following features:
* --vcf : specify required VCF file
* --bam : specify required BAM file (indexed with .bam.bai or .bai file)
* --subset : list of individual IDs to calculate the allele frequency. All individuals are used if unspecified
* --smID : If the individual ID in the BAM file and VCF file does not match, substitute the BAM file's ID into the specified argument
* --genoError : error rate of the external genotype file
* --minAF : minimum allele frequency of the markers to include
* --minAF : minimum call rate of the markers to include
* --site : If set, use only site information in the VCF and do not compare with the actual genotypes
* --self : Only compare the ID-matching individuals between the VCF and BAM file
* --best : Find the best matching individuals (.bestSM and .bestRG files will be produced). This option is substantially longer than the default option
* --free-none : Do not perform sequence-only method to estimate parameters
* --free-mix : (default) Estimate contamination using sequence-only method with Brent's single dimensional optimization.
* --free-refBias : Estimate the reference bias parameters using sequence-only method with Simplex method
* --free-full : Estimate both reference bias parameters and the contamination parameters using sequence-only method
* --chip-none : Do not perform sequence+array method to estimate parameters
* --free-mix : (default) Estimate contamination using sequence+array method with Brent's single dimensional optimization.
* --free-refBias : Estimate the refernece bias parameters using sequence+array method with Simplex method
* --free-full : Estimate both reference bias parameters and the contamination parameters using sequence+array method
* --ignoreRG : ignore the read grouup level comparison and compare samples only (recommended for an expedited run)
* --ignoreOverlapPair : ignore overlapping pair end fragment covering the same base. Disabling this option may decrease the sensitivity of the method when the insert size is short (with slight gain in the computational speed)
* --noEOF : do not check the EOF marker of the BAM file (for earlier version of BAM)
* --precise : calculate the likelihood in log-scale for high-depth data (recommended when --maxDepth is greater than 20. Can be a little bit slower)
* --minMapQ : minimum mapping quality of the sequence reads to compare
* --minQ : minimum base quality to include
* --maxQ : maximum base quality to cap
* --grid : the grid interval to search the optimum before running Brent's algorithm.
* --refRef : Initial Pr(refBase|HOMREFGeno) parameter
* --refHet : Initial Pr(refBase|HETGeno) parameter
* --refAlt : Initial Pr(refBase|HOMALTGeno) parameter
* --out : output file prefix (required)
* --verbose : print the progress of the method on the screeen
{{PhoneHomeParameters|hdr=====|bullet=1}}

== Principle of Operation ==

Each read group in a BAM file is evaluated independently. This means that in file with multiple read groups, problems will be flagged at the read group level (a plus). However, it also means that it might be hard to discern the correct assignment of read groups with very little data.

For each aligned base that overlaps a known genotype, we calculate the probability the probability that it was derived from a particular known genotype. This comparison considers only bases that overlap previously known genotypes and that meet the base quality and mapping quality thresholds.

Each individual in a pedigree has a different combination of genotypes, and bamGenotypeCheck will systematically search for the individual whose genotypes best match the observed read data.

For more about the technical details, see the page [[Verifying Sample Identities - Implementation]]

== Reference ==

Please cite the following paper:

G. Jun, M. Flickinger, K. N. Hetrick, Kurt, J. M. Romm, K. F. Doheny, G. Abecasis, M. Boehnke,and H. M. Kang, ''Detecting and Estimating Contamination of Human DNA Samples in Sequencing and Array-Based Genotype Data'', American journal of human genetics doi:10.1016/j.ajhg.2012.09.004 (volume 91 issue 5 pp.839 - 848)

== Contamination in Array Data ==

[[VerifyIDintensity]] or [[BAFRegress]] can estimate sample contamination from Illumina genotype array data.

== Acknowledgements ==

VerifyBamID is a result from collaborative effort by Hyun Min Kang, Goo Jun, Matthew Flickinger, Mary Kate Wing, Goncalo Abecasis, and Michael Boehnke. Please email to Hyun Min Kang [hmkang@umich.edu ] for any questions.

User talk:Gzajac

2017-01-04T19:42:09Z

Mktrost: Welcome!

'''Welcome to ''Genome Analysis Wiki''!'''
We hope you will contribute much and well.
You will probably want to read the [[Help:Contents|help pages]].
Again, welcome and have fun! [[User:Mktrost|Mktrost]] ([[User talk:Mktrost|talk]]) 14:42, 4 January 2017 (EST)

User:Gzajac

2017-01-04T19:42:08Z

Mktrost: Creating user page for new user.

University of Missouri BS Statistics 2009 - 2013
University of Michigan MS Biostatistics 2013 - 2015
University of Michigan PhD Biostatistics 2015 - In Progress

I have been in Goncalo’s lab since coming to the University of Michigan in 2013
I have been in the Genome Science Training Program since I started the PhD program in 2015

BamUtil: dedup

2016-04-15T02:30:35Z

Mktrost: /* Implementation Notes */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>dedup</code> function of <code>[[bamUtil]]</code> =
The <code>dedup</code> option of [[bamUtil]] determines duplicates in a coordinate sorted SAM/BAM file. It either marks or removes the lower quality duplicates.

This tool also contains the option to perform recalibration.

NOTE: This tool does not properly work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.

Potential future features:
* Soft clip overlapping reads (for now, use: [[BamUtil: clipOverlap]])

==Handling Duplicates==

The deduper reads all the alignments in a coordinate-sorted SAM/BAM looking for duplicates, failing if the file is not coordinate-sorted.

The deduper assumes that duplicates in the input BAM file are not marked. When the deduper detects a marked duplicate in the input BAM file, it will throw an error and stop. To override this behavior, use the [[#Ignore Previous Duplicate Marking (--force)|<code>--force</code>]] option; in this mode, alignments that are marked as duplicates in the input file are unmarked before the deduper begins its detection algorithm. The result is that only duplicates detected by the deduper will be marked in or removed from the output file.

The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate. If a mate is not found at the expected position, an error message is printed (once per file) indicating this error. Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads. Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate. If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.

With the default settings this tool should produce similar results as Picard.

Use the [[#Treat Reads with Mates On Different Chromosomes As Single-Ended (--oneChrom)|<code>--oneChrom</code>]] option to treat reads with a mate on a different chromosome as single-ended. This option is useful if you are running the deduper on just a single chromosome. The code will use less memory with this option if mates are found on different chromosomes. (Picard does not specially handle mates on different chromosomes, so the --oneChrom option may produce different results than Picard.)

=== Implementation Notes ===
Duplicates are determined by checking for matching keys.

The Key is comprised of:
# Chromosome
# Orientation (forward/reverse)
# Unclipped Start(forward)/End(reverse)
# Library

Rules:
* Skip Unmapped Reads, they are not marked as duplicate
* Reads whose mate is unmapped are treated as single-end
* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:
*# A paired-end record has the same key (even if the pair is not proper/the mate is not found) -OR-
*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])
* Mark both Paired-End Reads Duplicate if:
# Another paired-end pair has the same set of keys and has a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])

This code assumes that at most 1000 bases are clipped at the start of a read.

==Handling Recalibration==
See [[BamUtil: recab]] for recalibration details.

Recalibration parameters can be applied to deduping when --recab is specified.

== How to use it ==

When <code>dedup</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file is required, [[#input File (--in)|input File (--in)]], and must be sorted by coordinate.

The output SAM/BAM file is also required, [[#output File (--out)|output File (--out)]].

Recommended usage with Recalibrator:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Recalibrator:

/usr/cluster/bin/bam dedup --in ${INPUT}.bam --out ${OUTPUT}.bam --force --oneChrom

= Usage =
./bam dedup --in <InputBamFile> --out <OutputBamFile> [--minQual <minPhred>] [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--excludeFlags <flag>] [--verbose] [--noeof] [--params] [--recab]

Additional Recalibration Usage is documented at [[BamUtil: recab#Usage|BamUtil: recab -> Usage]]

= Parameters =
<pre>
Required parameters :
--in <infile> : Input BAM file name (must be sorted)
--out <outfile> : Output BAM file name (same order with original file)
Optional parameters :
--minQual <int> : Only add scores over this phred quality when determining a read's quality (default: 15)
--log <logfile> : Log and summary statistics (default: [outfile].log, or stderr if --out starts with '-')
--oneChrom : Treat reads with mates on different chromosomes as single-ended.
--rmDups : Remove duplicates (default is to mark duplicates)
--force : Allow an already mark-duplicated BAM file, unmarking any previously marked
duplicates and apply this duplicate marking logic. Default is to throw errors
and exit when trying to run on an already mark-duplicated BAM
--excludeFlags <flag> : exclude reads with any of these flags set when determining or marking duplicates
by default (0xB04): exclude unmapped, secondary reads, QC failures, and supplementary reads
--verbose : Turn on verbose mode
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings
--recab : Recalibrate in addition to deduping
</pre>
{{PhoneHomeParamDesc}}

Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters|BamUtil: recab -> Parameters]]

== Required Parameters ==
{{inBAMInputFile}}

Note: The input file must be sorted by coordinate.
{{outBAMOutputFile}}

== Optional Parameters==
=== Minimum Quality for Quality Calculations (<code>--minQual</code>) ===

When duplicate reads are encountered, the read with the highest quality is kept.

To determine the quality of a read, all of the phred base quality scores above the <code>--minQual</code> value are added together. If <code>--minQual</code> is not specified, it is defaulted to 15.

=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Treat Reads with Mates On Different Chromosomes As Single-Ended (<code>--oneChrom</code>) ===

If a read's mate is not found it will not be used for duplicate marking. If you are running on a single chromosome, all read's whose mates are on different chromosomes will not be used for duplicate marking. The <code>--oneChrom</code> option will treat reads with mates on a different chromosome as single-ended.

=== Remove Duplicates (<code>--rmDups</code>) ===

Instead of marking a read as duplicate in the flag, the <code>--rmDups</code> option will remove it from the output BAM file.

=== Ignore Previous Duplicate Marking (<code>--force</code>) ===

By default the deduper will throw an error and stop if a read is already marked as duplicate. The <code>--force</code> option will removes any previous duplicate marking and marks the reads from scratch. The resulting output file will only have reads determined by the deduper marked as duplicates.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0xB04

By default skips reads with any of the following flags set:
* unmapped
* secondary alignment
* fails QC checks
* supplementary reads

Secondary (0x100) and Supplementary (0x800) reads currently must be excluded.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

== Recalibrate (<code>--recab</code>) ==

This option will recalibrate the input file in addition to deduping.

See [[BamUtil: recab]] for recalibration details.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: gapInfo

2016-03-06T05:45:28Z

Mktrost: /* Rules */

[[Category:BamUtil|gapInfo]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>gapInfo</code> function of <code>bamUtil</code> =
The <code>gapInfo</code> option on the [[bamUtil]] prints information on the gap between read pairs in a SAM/BAM File.

There are two ways to run: standard/detailed. To run as detailed, use the <code>--detailed</code> option.

Standard output prints the number of pairs that have a given gap size.

The gap size is calculated by counting the number of bases between the clipped end of the first read and the clipped start of the 2nd read. (mate0BasedClippedStart - 0BasedPositionClippedEnd - 1)

The gap size will be negative if the reads overlap.

= Rules =
gapInfo skips any records that are marked in the flag as:
* unmapped
* not paired
* mate is unmapped
* secondary alignment (not primary alignment)
* supplementary alignment
* duplicates
* QC Failure
* mate is on a different chromosome
* chromosome is unknown (-1/*)
* mate starts before this record
* mate starts at the same location as this record & this record is the reverse strand
* reverse strands (unless <code>--detailed</code> is specified)

When <code>--refFile</code> and <code>--detailed</code> is not specified gaps that contain reference base 'N' are skipped.

= Usage =
./bam gapInfo --in <inputFile> --out <outputFile> [--noeof] [--params]

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to print read pair gap info for
--out : the output file to be written
Optional Parameters:
--refFile : reference file, used to skip gaps that include reference base 'N' (for runs without --detailed) --detailed : Print the details for each read pair
Optional Parameters for the Detailed Option:
--checkFirst : Check the first in pair flag and print "NotFirst" if it isn't first
--checkStrand : Check the strand flag and print "Reverse" if it is reverse complimented
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings to stderr
</pre>
{{PhoneHomeParamDesc}}

== Required Parameters ==
{{inBAMInputFile}}

=== Output File (<code>--out</code>) ===
Use <code>--out</code> followed by a file name to specify the output file to write.

The Standard Output prints a 2-column (separated by tabs) line for each gapSize found in the SAM/BAM file. The first column contains the gap size and the 2nd column contains the number of pairs that have that gap size. The first line is a header line describing the columns.

Detailed output does not have a header line and is described below under the [[#Print Detailed Per-Pair Information (<code>--detailed</code>) | --detailed]] parameter.

== Optional Prameters ==
{{RefFile}}

With this option specified, do not increment counters for the number of times that a gap is found if any of the reference bases in the gap are an 'N'. (N/A if <code>--detailed</code> is specified.)

=== Print Detailed Per-Pair Information (<code>--detailed</code>) ===
With this option, for every record processed per the above rules, the following information is printed on a line as tab separated columns:
* Reference/Chromosome Name
* 1-based read end position (clipped)
* gap size
Additional columns if <code>--checkFirst</code> and/or <code>--checkStrand</code> are specified.

Detailed output does not have a header line.

See [[#Optional Parameters for --detailed|Optional Parameters for <code>--detailed</code>]] for additional options related to <code>--detailed</code>.

{{noeofBGZFParameter}}
{{paramsParameter}}

== Optional Parameters for [[#Print Detailed Per-Pair Information (--detailed)|<code>--detailed</code>]] ==
=== Check First (<code>--checkFirst</code>) ===
Only applicable if [[#Print Detailed Per-Pair Information (--detailed)|<code>--detailed</code>]] is also provided.

When specified along with [[#Print Detailed Per-Pair Information (--detailed)|<code>--detailed</code>]], the output for each record processed also includes "NotFirst" if it is not marked as FirstFragment in the flags.

=== Check Strand (<code>--checkStrand</code>) ===
Only applicable if [[#Print Detailed Per-Pair Information (--detailed)|<code>--detailed</code>]] is also provided.

When specified along with [[#Print Detailed Per-Pair Information (--detailed)|<code>--detailed</code>]], the output for each record processed also includes "Reverse" if it is marked as the reverse strand in the flags.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

= Output =

All status messages are written to stderr.

Tab-delimited columns as described above.

== Example Output ==
For standard output:
<pre>
GapSize NumPairs
-23 3
-21 3
-20 4
-5 1
30 1
70 3
</pre>

For detailed output with both <code>--checkFirst</code> & <code>--checkStrand</code> specified:
<pre>
1 28 70
1 10028 71 NotFirst Reverse
1 10028 70
1 10028 70
1 10028 30
1 10028 -19 NotFirst Reverse
1 10028 -19 NotFirst Reverse
1 10028 -19 NotFirst Reverse
1 10030 -21
1 10030 -20
1 10030 -20
1 10030 -21
1 10030 -21
1 10030 -20
1 10030 -20
2 32 -18 NotFirst Reverse
4 24 -23
4 27 -23
4 30 -23
4 34 -5
</pre>

BamUtil: recab

2016-03-06T05:10:36Z

Mktrost: /* Parameters */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>recab</code> function of <code>[[bamUtil]]</code> =
The <code>recab</code> option of [[bamUtil]] recalibrates a SAM/BAM file.

Recalibration can also be called as an option of [[bamUtil: dedup]]. This will perform the recalibration and the deduping in the same set of steps, increasing processing speed.

==Handling Recalibration/Implementation Notes==

Recalibration is a 2-step process that loops through the file twice:
# Build Recalibration Table
# Apply Recalibration Table

The Recalibration Table groups bases based on a set of covariates:
* Read Group
* Quality (either from the quality string or [[#Read the quality from a tag (--qualField)|from a tag]])
* Cycle (reverse complement for reverse strands)
* 1st/2nd read in pair
* Previous Cycle's Base (reverse complement for reverse strands)
* This Cycle's Base (reverse complement for reverse strands)

The Recalibration Table tracks the number of matches/mismatches for each set of covariates.

Only bases meeting all of the following criteria are used to Build the Recalibration Table:
* Read criteria
** not a duplicate
** mapped
** mapping quality != 0
** mapping quality != 255
* Base criteria
** match/mismatch (not an insertion/deletion/skip/clip)
** not a [[#DBSNP File (--dbsnp)|dbSNP position]]
** base quality > [[#Minimum Recalibration Base Quality (--minBaseQual)|minBaseQual (5 by default)]]
* Additional criteria for cycle != 1 (can be turned off via flags)
** previous base is a CIGAR Match/Mismatch (Use [[#Allow Previous Base Non-Match/Mismatch (--keepPrevNonAdjacent)|<code>--keepPrevNonAdjacent</code>]] to disable)
** previous base position is not a [[#DBSNP File (--dbsnp)|dbSNP position]] (Use [[#Allow Previous Base DBSNP (--keepPrevDbsnp)|<code>--keepPrevDbsnp</code>]] to disable)

The Recalibration Table is applied to all bases meeting all of the following criteria (even if they were not used for creating the table):
* base quality > [[#Minimum Recalibration Base Quality (--minBaseQual)|minBaseQual (5 by default)]]
* at least 1 match or mismatch for the set of covariates

Recalibrated Quality is: <math>-10 * \log \frac{mismatches + 1}{mismatches + matches + 1}</math>

Alternatively, [[#Logistic Regression (--useLogReg)|logistic regression]] can be used for calculating the new quality.

If the Recalibrated Quality is greater than [[#Maximum Recalibration Base Quality (--maxBaseQual)|maxBaseQual]], the updated quality is set to maxBaseQual.

Optionally, the previous quality can be [[#Store the original quality (--storeQualTag)|stored in a tag]].

The current recalibration logic was designed for recalibrating ILLUMINA data.

NOTE: GATK ignores/skips adapters, but our logic does not.

== How to use it ==

When <code>recab</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file ([[#input File (--in)|--in]]), the output SAM/BAM file ([[#output File (--out)|--out]]), and the reference file ([[#Reference File (--refFile)|--refFile]]) are required inputs.

Recommended usage with Deduper:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Deduper:

/usr/cluster/bin/bam recab --in ${INPUT}.bam --out ${OUTPUT}.bam --refFile ${REF} --dbsnp ${DBSNP} --storeQualTag OQ --maxBaseQual 40

= Usage =
./bam recab (options) --in <InputBamFile> --out <OutputFile> [--log <logFile>] [--verbose] [--noeof] [--params] --refFile <ReferenceFile> [--dbsnp <dbsnpFile>] [--minBaseQual <minBaseQual>] [--maxBaseQual <maxBaseQual>] [--blended <weight>] [--fitModel] [--fast] [--keepPrevDbsnp] [--keepPrevNonAdjacent] [--useLogReg] [--qualField <tag>] [--storeQualTag <tag>] [--buildExcludeFlags <flag>] [--applyExcludeFlags <flag>]

= Parameters =
<pre>
Required General Parameters :
--in <infile> : input BAM file name
--out <outfile> : output recalibration file name
Optional General Parameters :
--log <logfile> : log and summary statistics (default: [outfile].log)
--verbose : Turn on verbose mode
--noeof : do not expect an EOF block on a bam file.
--params : print the parameter settings

Recab Specific Required Parameters
--refFile <reference file> : reference file name
Recab Specific Optional Parameters :
--dbsnp <known variance file> : dbsnp file of positions
--minBaseQual <minBaseQual> : minimum base quality of bases to recalibrate (default: 5)
--maxBaseQual <maxBaseQual> : maximum recalibrated base quality (default: 50)
qualities over this value will be set to this value.
This setting is applied after binning (if applicable).
--blended <weight> : blended model weight
--fitModel : check if the logistic regression model fits the data
overriden by fast, but automatically applied by useLogReg
--fast : use a compact representation that only allows:
* at most 256 Read Groups
* maximum quality 63
* at most 127 cycles
overrides fitModel, but is overridden by useLogReg
uses up to about 2.25G more memory than running without --fast.
--keepPrevDbsnp : do not exclude entries where the previous base is in dbsnp when
building the recalibration table
By default they are excluded from the table.
--keepPrevNonAdjacent : do not exclude entries where the previous base is not adjacent
(not a Cigar M/X/=) when building the recalibration table
By default they are excluded from the table (except the first cycle).
--useLogReg : use logistic regression calculated quality for the new quality
automatically applies fitModel and overrides fast.
--qualField <quality tag> : tag to get the starting base quality
(default is to get it from the Quality field)
--storeQualTag <quality tag> : tag to store the previous quality into
--buildExcludeFlags <flag> : exclude reads with any of these flags set when building the
recalibration table. Default is 0xF04
--applyExcludeFlags <flag> : do not apply the recalibration table to any reads with any of these flags set
Quality Binning Parameters (optional):
Bin qualities by phred score, into the ranges specified by binQualS or binQualF (both cannot be used)
Ranges are specified by comma separated minimum phred score for the bin, example: 1,17,20,30,40,50,70
The first bin always starts at 0, so does not need to be specified.
By default, the bin value is the low end of the range.
--binQualS : Bin the Qualities as specified (phred): minQualOfBin2, minQualofBin3...
--binQualF : Bin the Qualities based on the specified file
--binMid : Use the mid point of the quality bin range for the quality value of the bin.
--binHigh : Use the high end of the quality bin range for the quality value of the bin.

</pre>
{{PhoneHomeParamDesc}}

== Required Generic Parameters ==
{{inBAMInputFile}}
{{outBAMOutputFile}}

== Optional Generic Parameters ==
=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

{{PhoneHomeParameters}}

== Required Recalibration Parameters ==
=== Reference File (<code>--refFile</code>) ===

The reference file is a required parameter used for comparing read bases to the reference.

== Optional Recalibration Parameters ==

=== DBSNP File (<code>--dbsnp</code>) ===

The dbsnp file that specifies positions to skip recalibrating. Tab delimited file with the chromosome in the first column and the 1-based position in the 2nd column.

=== Minimum Recalibration Base Quality (<code>--minBaseQual</code>) ===

When recalibrating reads, only positions with a base quality greater than this minimum phred quality will be recalibrated. If <code>--minBaseQual</code> is not specified, it is defaulted to 5.

The ILLUMINA specs indicate that any quality below 5 can be used as an error indicator so we do not want to recalibrate those.

=== Maximum Recalibration Base Quality (<code>--maxBaseQual</code>) ===

This value sets the maximum phred base quality assigned to a base after recalibrating. Any qualities above this value will be set to this value. It is defaulted to 50.

=== Blended Model Weight (<code>--blended</code>) ===

TBD - this parameter is not yet implemented.

=== Fit Model (<code>--fitModel</code>) ===

Check if the logistic regression model fits the data.

This option does NOT set the new qualities to the logistic regression calculated qualities, it only checks the fit. To apply the logistic regression qualities, see [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]]. <code>--fitModel</code> is automatically applied when <code>--useLogReg</code> is specified.

This option cannot be used in conjunction with [[#Fast Recalibration (--fast)|<code>--fast</code>]] and is overriden by <code>--fast</code>, but automatically applied by useLogReg

=== Fast Recalibration (<code>--fast</code>) ===

Use a compact representation of the Recalibration Table that only allows:
* at most 256 Read Groups
* maximum quality 63
* at most 127 cycles

This option will run faster than the default recalibration, but uses up to about 2.25G more memory than running without --fast.

This option cannot be used in conjunction with [[#Fit Model (--fitModel)|<code>--fitModel</code>]], or [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]] and overrides [[#Fit Model (--fitModel)|<code>--fitModel</code>]], but is overridden by [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]].

=== Allow Previous Base DBSNP (<code>--keepPrevDbsnp</code>) ===

By default bases where the previous base is in DBSNP are excluded from the Recalibration Table.

This option includes these bases in the building of the Recalibration Table.

=== Allow Previous Base Non-Match/Mismatch (<code>--keepPrevNonAdjacent</code>) ===

By default bases where the previous base is not a CIGAR Match/Mismatch are excluded from the Recalibration Table.

This option includes these bases in the building of the Recalibration Table.

=== Logistic Regression (<code>--useLogReg</code>) ===

Use the logistic regression empirical qualities for setting the new base qualities instead of the default formula.

This option automatically enables [[#Fit Model (--fitModel)|<code>--fitModel</code>]] and disables [[#Fast Recalibration (--fast)|<code>--fast</code>]].

=== Read the quality from a tag (<code>--qualField</code>) ===

If this parameter is set, then read the quality string from the specified tag name. If the tag is not found, the quality is read from the quality field.

=== Store the original quality (<code>--storeQualTag</code>) ===

If this parameter is set, the original quality will be stored as a string in the specified tag.

=== Skip Records with any of the Specified Flags (<code>--buildExcludeFlags</code>, <code>--applyExcludeFlags</code>) ===
Use <code>--buildExcludeFlags</code> to skip records with any of the specified flags set when building the recalibration table, default 0xF04.

By default, when building the recalibration table reads with any of the following flags set are skipped:
* unmapped
* secondary alignment
* fails QC checks
* duplicate
* supplementary alignment

Use <code>--applyExcludeFlags</code> to skip records with any of the specified flags set when applying the recalibration table. The default value is 0x000, do not skip any reads.

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: recab

2016-03-06T05:07:51Z

Mktrost: /* Skip Records with any of the Specified Flags (--buildExcludeFlags, --applyExcludeFlags) */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>recab</code> function of <code>[[bamUtil]]</code> =
The <code>recab</code> option of [[bamUtil]] recalibrates a SAM/BAM file.

Recalibration can also be called as an option of [[bamUtil: dedup]]. This will perform the recalibration and the deduping in the same set of steps, increasing processing speed.

==Handling Recalibration/Implementation Notes==

Recalibration is a 2-step process that loops through the file twice:
# Build Recalibration Table
# Apply Recalibration Table

The Recalibration Table groups bases based on a set of covariates:
* Read Group
* Quality (either from the quality string or [[#Read the quality from a tag (--qualField)|from a tag]])
* Cycle (reverse complement for reverse strands)
* 1st/2nd read in pair
* Previous Cycle's Base (reverse complement for reverse strands)
* This Cycle's Base (reverse complement for reverse strands)

The Recalibration Table tracks the number of matches/mismatches for each set of covariates.

Only bases meeting all of the following criteria are used to Build the Recalibration Table:
* Read criteria
** not a duplicate
** mapped
** mapping quality != 0
** mapping quality != 255
* Base criteria
** match/mismatch (not an insertion/deletion/skip/clip)
** not a [[#DBSNP File (--dbsnp)|dbSNP position]]
** base quality > [[#Minimum Recalibration Base Quality (--minBaseQual)|minBaseQual (5 by default)]]
* Additional criteria for cycle != 1 (can be turned off via flags)
** previous base is a CIGAR Match/Mismatch (Use [[#Allow Previous Base Non-Match/Mismatch (--keepPrevNonAdjacent)|<code>--keepPrevNonAdjacent</code>]] to disable)
** previous base position is not a [[#DBSNP File (--dbsnp)|dbSNP position]] (Use [[#Allow Previous Base DBSNP (--keepPrevDbsnp)|<code>--keepPrevDbsnp</code>]] to disable)

The Recalibration Table is applied to all bases meeting all of the following criteria (even if they were not used for creating the table):
* base quality > [[#Minimum Recalibration Base Quality (--minBaseQual)|minBaseQual (5 by default)]]
* at least 1 match or mismatch for the set of covariates

Recalibrated Quality is: <math>-10 * \log \frac{mismatches + 1}{mismatches + matches + 1}</math>

Alternatively, [[#Logistic Regression (--useLogReg)|logistic regression]] can be used for calculating the new quality.

If the Recalibrated Quality is greater than [[#Maximum Recalibration Base Quality (--maxBaseQual)|maxBaseQual]], the updated quality is set to maxBaseQual.

Optionally, the previous quality can be [[#Store the original quality (--storeQualTag)|stored in a tag]].

The current recalibration logic was designed for recalibrating ILLUMINA data.

NOTE: GATK ignores/skips adapters, but our logic does not.

== How to use it ==

When <code>recab</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file ([[#input File (--in)|--in]]), the output SAM/BAM file ([[#output File (--out)|--out]]), and the reference file ([[#Reference File (--refFile)|--refFile]]) are required inputs.

Recommended usage with Deduper:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Deduper:

/usr/cluster/bin/bam recab --in ${INPUT}.bam --out ${OUTPUT}.bam --refFile ${REF} --dbsnp ${DBSNP} --storeQualTag OQ --maxBaseQual 40

= Usage =
./bam recab (options) --in <InputBamFile> --out <OutputFile> [--log <logFile>] [--verbose] [--noeof] [--params] --refFile <ReferenceFile> [--dbsnp <dbsnpFile>] [--minBaseQual <minBaseQual>] [--maxBaseQual <maxBaseQual>] [--blended <weight>] [--fitModel] [--fast] [--keepPrevDbsnp] [--keepPrevNonAdjacent] [--useLogReg] [--qualField <tag>] [--storeQualTag <tag>] [--buildExcludeFlags <flag>] [--applyExcludeFlags <flag>]

= Parameters =
<pre>
Required General Parameters :
--in <infile> : input BAM file name
--out <outfile> : output recalibration file name
Optional General Parameters :
--log <logfile> : log and summary statistics (default: [outfile].log)
--verbose : Turn on verbose mode
--noeof : do not expect an EOF block on a bam file.
--params : print the parameter settings

Recab Specific Required Parameters
--refFile <reference file> : reference file name
Recab Specific Optional Parameters :
--dbsnp <known variance file> : dbsnp file of positions
--minBaseQual <minBaseQual> : minimum base quality of bases to recalibrate (default: 5)
--maxBaseQual <maxBaseQual> : maximum recalibrated base quality (default: 50)
--blended <weight> : blended model weight
--fitModel : check if the logistic regression model fits the data
overriden by fast, but automatically applied by useLogReg
--fast : use a compact representation that only allows:
* at most 256 Read Groups
* maximum quality 63
* at most 127 cycles
overrides fitModel, but is overridden by useLogReg
uses up to about 2.25G more memory than running without --fast.
--keepPrevDbsnp : do not exclude entries where the previous base is in dbsnp when
building the recalibration table
By default they are excluded from the table.
--keepPrevNonAdjacent : do not exclude entries where the previous base is not adjacent
(not a Cigar M/X/=) when building the recalibration table
By default they are excluded from the table (except the first cycle).
--useLogReg : use logistic regression calculated quality for the new quality
automatically applies fitModel and overrides fast.
--qualField <quality tag> : tag to get the starting base quality
(default is to get it from the Quality field)
--storeQualTag <quality tag> : tag to store the previous quality into
--buildExcludeFlags <flag> : exclude reads with any of these flags set when building the
recalibration table
--applyExcludeFlags <flag> : do not apply the recalibration table to any reads with any of these flags set
</pre>
{{PhoneHomeParamDesc}}

== Required Generic Parameters ==
{{inBAMInputFile}}
{{outBAMOutputFile}}

== Optional Generic Parameters ==
=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

{{PhoneHomeParameters}}

== Required Recalibration Parameters ==
=== Reference File (<code>--refFile</code>) ===

The reference file is a required parameter used for comparing read bases to the reference.

== Optional Recalibration Parameters ==

=== DBSNP File (<code>--dbsnp</code>) ===

The dbsnp file that specifies positions to skip recalibrating. Tab delimited file with the chromosome in the first column and the 1-based position in the 2nd column.

=== Minimum Recalibration Base Quality (<code>--minBaseQual</code>) ===

When recalibrating reads, only positions with a base quality greater than this minimum phred quality will be recalibrated. If <code>--minBaseQual</code> is not specified, it is defaulted to 5.

The ILLUMINA specs indicate that any quality below 5 can be used as an error indicator so we do not want to recalibrate those.

=== Maximum Recalibration Base Quality (<code>--maxBaseQual</code>) ===

This value sets the maximum phred base quality assigned to a base after recalibrating. Any qualities above this value will be set to this value. It is defaulted to 50.

=== Blended Model Weight (<code>--blended</code>) ===

TBD - this parameter is not yet implemented.

=== Fit Model (<code>--fitModel</code>) ===

Check if the logistic regression model fits the data.

This option does NOT set the new qualities to the logistic regression calculated qualities, it only checks the fit. To apply the logistic regression qualities, see [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]]. <code>--fitModel</code> is automatically applied when <code>--useLogReg</code> is specified.

This option cannot be used in conjunction with [[#Fast Recalibration (--fast)|<code>--fast</code>]] and is overriden by <code>--fast</code>, but automatically applied by useLogReg

=== Fast Recalibration (<code>--fast</code>) ===

Use a compact representation of the Recalibration Table that only allows:
* at most 256 Read Groups
* maximum quality 63
* at most 127 cycles

This option will run faster than the default recalibration, but uses up to about 2.25G more memory than running without --fast.

This option cannot be used in conjunction with [[#Fit Model (--fitModel)|<code>--fitModel</code>]], or [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]] and overrides [[#Fit Model (--fitModel)|<code>--fitModel</code>]], but is overridden by [[#Logistic Regression (--useLogReg)|<code>--useLogReg</code>]].

=== Allow Previous Base DBSNP (<code>--keepPrevDbsnp</code>) ===

By default bases where the previous base is in DBSNP are excluded from the Recalibration Table.

This option includes these bases in the building of the Recalibration Table.

=== Allow Previous Base Non-Match/Mismatch (<code>--keepPrevNonAdjacent</code>) ===

By default bases where the previous base is not a CIGAR Match/Mismatch are excluded from the Recalibration Table.

This option includes these bases in the building of the Recalibration Table.

=== Logistic Regression (<code>--useLogReg</code>) ===

Use the logistic regression empirical qualities for setting the new base qualities instead of the default formula.

This option automatically enables [[#Fit Model (--fitModel)|<code>--fitModel</code>]] and disables [[#Fast Recalibration (--fast)|<code>--fast</code>]].

=== Read the quality from a tag (<code>--qualField</code>) ===

If this parameter is set, then read the quality string from the specified tag name. If the tag is not found, the quality is read from the quality field.

=== Store the original quality (<code>--storeQualTag</code>) ===

If this parameter is set, the original quality will be stored as a string in the specified tag.

=== Skip Records with any of the Specified Flags (<code>--buildExcludeFlags</code>, <code>--applyExcludeFlags</code>) ===
Use <code>--buildExcludeFlags</code> to skip records with any of the specified flags set when building the recalibration table, default 0xF04.

By default, when building the recalibration table reads with any of the following flags set are skipped:
* unmapped
* secondary alignment
* fails QC checks
* duplicate
* supplementary alignment

Use <code>--applyExcludeFlags</code> to skip records with any of the specified flags set when applying the recalibration table. The default value is 0x000, do not skip any reads.

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: dedup

2016-03-06T04:45:48Z

Mktrost: /* Skip Records with any of the Specified Flags (--excludeFlags) */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>dedup</code> function of <code>[[bamUtil]]</code> =
The <code>dedup</code> option of [[bamUtil]] determines duplicates in a coordinate sorted SAM/BAM file. It either marks or removes the lower quality duplicates.

This tool also contains the option to perform recalibration.

NOTE: This tool does not properly work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.

Potential future features:
* Soft clip overlapping reads (for now, use: [[BamUtil: clipOverlap]])

==Handling Duplicates==

The deduper reads all the alignments in a coordinate-sorted SAM/BAM looking for duplicates, failing if the file is not coordinate-sorted.

The deduper assumes that duplicates in the input BAM file are not marked. When the deduper detects a marked duplicate in the input BAM file, it will throw an error and stop. To override this behavior, use the [[#Ignore Previous Duplicate Marking (--force)|<code>--force</code>]] option; in this mode, alignments that are marked as duplicates in the input file are unmarked before the deduper begins its detection algorithm. The result is that only duplicates detected by the deduper will be marked in or removed from the output file.

The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate. If a mate is not found at the expected position, an error message is printed (once per file) indicating this error. Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads. Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate. If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.

With the default settings this tool should produce similar results as Picard.

Use the [[#Treat Reads with Mates On Different Chromosomes As Single-Ended (--oneChrom)|<code>--oneChrom</code>]] option to treat reads with a mate on a different chromosome as single-ended. This option is useful if you are running the deduper on just a single chromosome. The code will use less memory with this option if mates are found on different chromosomes. (Picard does not specially handle mates on different chromosomes, so the --oneChrom option may produce different results than Picard.)

=== Implementation Notes ===
Duplicates are determined by checking for matching keys.

The Key is comprised of:
# Chromosome
# Orientation (forward/reverse)
# Unclipped Start(forward)/End(reverse)
# Library

Rules:
* Skip Unmapped Reads, they are not marked as duplicate
* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:
*# A paired-end record has the same key (even if the pair is not proper/the mate is unmapped/the mate is not found) -OR-
*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])
* Mark both Paired-End Reads Duplicate if:
# Another paired-end pair has the same set of keys and has a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])

This code assumes that at most 1000 bases are clipped at the start of a read.

==Handling Recalibration==
See [[BamUtil: recab]] for recalibration details.

Recalibration parameters can be applied to deduping when --recab is specified.

== How to use it ==

When <code>dedup</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file is required, [[#input File (--in)|input File (--in)]], and must be sorted by coordinate.

The output SAM/BAM file is also required, [[#output File (--out)|output File (--out)]].

Recommended usage with Recalibrator:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Recalibrator:

/usr/cluster/bin/bam dedup --in ${INPUT}.bam --out ${OUTPUT}.bam --force --oneChrom

= Usage =
./bam dedup --in <InputBamFile> --out <OutputBamFile> [--minQual <minPhred>] [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--excludeFlags <flag>] [--verbose] [--noeof] [--params] [--recab]

Additional Recalibration Usage is documented at [[BamUtil: recab#Usage|BamUtil: recab -> Usage]]

= Parameters =
<pre>
Required parameters :
--in <infile> : Input BAM file name (must be sorted)
--out <outfile> : Output BAM file name (same order with original file)
Optional parameters :
--minQual <int> : Only add scores over this phred quality when determining a read's quality (default: 15)
--log <logfile> : Log and summary statistics (default: [outfile].log, or stderr if --out starts with '-')
--oneChrom : Treat reads with mates on different chromosomes as single-ended.
--rmDups : Remove duplicates (default is to mark duplicates)
--force : Allow an already mark-duplicated BAM file, unmarking any previously marked
duplicates and apply this duplicate marking logic. Default is to throw errors
and exit when trying to run on an already mark-duplicated BAM
--excludeFlags <flag> : exclude reads with any of these flags set when determining or marking duplicates
by default (0xB04): exclude unmapped, secondary reads, QC failures, and supplementary reads
--verbose : Turn on verbose mode
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings
--recab : Recalibrate in addition to deduping
</pre>
{{PhoneHomeParamDesc}}

Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters|BamUtil: recab -> Parameters]]

== Required Parameters ==
{{inBAMInputFile}}

Note: The input file must be sorted by coordinate.
{{outBAMOutputFile}}

== Optional Parameters==
=== Minimum Quality for Quality Calculations (<code>--minQual</code>) ===

When duplicate reads are encountered, the read with the highest quality is kept.

To determine the quality of a read, all of the phred base quality scores above the <code>--minQual</code> value are added together. If <code>--minQual</code> is not specified, it is defaulted to 15.

=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Treat Reads with Mates On Different Chromosomes As Single-Ended (<code>--oneChrom</code>) ===

If a read's mate is not found it will not be used for duplicate marking. If you are running on a single chromosome, all read's whose mates are on different chromosomes will not be used for duplicate marking. The <code>--oneChrom</code> option will treat reads with mates on a different chromosome as single-ended.

=== Remove Duplicates (<code>--rmDups</code>) ===

Instead of marking a read as duplicate in the flag, the <code>--rmDups</code> option will remove it from the output BAM file.

=== Ignore Previous Duplicate Marking (<code>--force</code>) ===

By default the deduper will throw an error and stop if a read is already marked as duplicate. The <code>--force</code> option will removes any previous duplicate marking and marks the reads from scratch. The resulting output file will only have reads determined by the deduper marked as duplicates.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0xB04

By default skips reads with any of the following flags set:
* unmapped
* secondary alignment
* fails QC checks
* supplementary reads

Secondary (0x100) and Supplementary (0x800) reads currently must be excluded.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

== Recalibrate (<code>--recab</code>) ==

This option will recalibrate the input file in addition to deduping.

See [[BamUtil: recab]] for recalibration details.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: dedup

2016-03-06T04:20:58Z

Mktrost: /* Skip Records with any of the Specified Flags (--excludeFlags) */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>dedup</code> function of <code>[[bamUtil]]</code> =
The <code>dedup</code> option of [[bamUtil]] determines duplicates in a coordinate sorted SAM/BAM file. It either marks or removes the lower quality duplicates.

This tool also contains the option to perform recalibration.

NOTE: This tool does not properly work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.

Potential future features:
* Soft clip overlapping reads (for now, use: [[BamUtil: clipOverlap]])

==Handling Duplicates==

The deduper reads all the alignments in a coordinate-sorted SAM/BAM looking for duplicates, failing if the file is not coordinate-sorted.

The deduper assumes that duplicates in the input BAM file are not marked. When the deduper detects a marked duplicate in the input BAM file, it will throw an error and stop. To override this behavior, use the [[#Ignore Previous Duplicate Marking (--force)|<code>--force</code>]] option; in this mode, alignments that are marked as duplicates in the input file are unmarked before the deduper begins its detection algorithm. The result is that only duplicates detected by the deduper will be marked in or removed from the output file.

The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate. If a mate is not found at the expected position, an error message is printed (once per file) indicating this error. Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads. Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate. If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.

With the default settings this tool should produce similar results as Picard.

Use the [[#Treat Reads with Mates On Different Chromosomes As Single-Ended (--oneChrom)|<code>--oneChrom</code>]] option to treat reads with a mate on a different chromosome as single-ended. This option is useful if you are running the deduper on just a single chromosome. The code will use less memory with this option if mates are found on different chromosomes. (Picard does not specially handle mates on different chromosomes, so the --oneChrom option may produce different results than Picard.)

=== Implementation Notes ===
Duplicates are determined by checking for matching keys.

The Key is comprised of:
# Chromosome
# Orientation (forward/reverse)
# Unclipped Start(forward)/End(reverse)
# Library

Rules:
* Skip Unmapped Reads, they are not marked as duplicate
* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:
*# A paired-end record has the same key (even if the pair is not proper/the mate is unmapped/the mate is not found) -OR-
*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])
* Mark both Paired-End Reads Duplicate if:
# Another paired-end pair has the same set of keys and has a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])

This code assumes that at most 1000 bases are clipped at the start of a read.

==Handling Recalibration==
See [[BamUtil: recab]] for recalibration details.

Recalibration parameters can be applied to deduping when --recab is specified.

== How to use it ==

When <code>dedup</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file is required, [[#input File (--in)|input File (--in)]], and must be sorted by coordinate.

The output SAM/BAM file is also required, [[#output File (--out)|output File (--out)]].

Recommended usage with Recalibrator:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Recalibrator:

/usr/cluster/bin/bam dedup --in ${INPUT}.bam --out ${OUTPUT}.bam --force --oneChrom

= Usage =
./bam dedup --in <InputBamFile> --out <OutputBamFile> [--minQual <minPhred>] [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--excludeFlags <flag>] [--verbose] [--noeof] [--params] [--recab]

Additional Recalibration Usage is documented at [[BamUtil: recab#Usage|BamUtil: recab -> Usage]]

= Parameters =
<pre>
Required parameters :
--in <infile> : Input BAM file name (must be sorted)
--out <outfile> : Output BAM file name (same order with original file)
Optional parameters :
--minQual <int> : Only add scores over this phred quality when determining a read's quality (default: 15)
--log <logfile> : Log and summary statistics (default: [outfile].log, or stderr if --out starts with '-')
--oneChrom : Treat reads with mates on different chromosomes as single-ended.
--rmDups : Remove duplicates (default is to mark duplicates)
--force : Allow an already mark-duplicated BAM file, unmarking any previously marked
duplicates and apply this duplicate marking logic. Default is to throw errors
and exit when trying to run on an already mark-duplicated BAM
--excludeFlags <flag> : exclude reads with any of these flags set when determining or marking duplicates
by default (0xB04): exclude unmapped, secondary reads, QC failures, and supplementary reads
--verbose : Turn on verbose mode
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings
--recab : Recalibrate in addition to deduping
</pre>
{{PhoneHomeParamDesc}}

Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters|BamUtil: recab -> Parameters]]

== Required Parameters ==
{{inBAMInputFile}}

Note: The input file must be sorted by coordinate.
{{outBAMOutputFile}}

== Optional Parameters==
=== Minimum Quality for Quality Calculations (<code>--minQual</code>) ===

When duplicate reads are encountered, the read with the highest quality is kept.

To determine the quality of a read, all of the phred base quality scores above the <code>--minQual</code> value are added together. If <code>--minQual</code> is not specified, it is defaulted to 15.

=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Treat Reads with Mates On Different Chromosomes As Single-Ended (<code>--oneChrom</code>) ===

If a read's mate is not found it will not be used for duplicate marking. If you are running on a single chromosome, all read's whose mates are on different chromosomes will not be used for duplicate marking. The <code>--oneChrom</code> option will treat reads with mates on a different chromosome as single-ended.

=== Remove Duplicates (<code>--rmDups</code>) ===

Instead of marking a read as duplicate in the flag, the <code>--rmDups</code> option will remove it from the output BAM file.

=== Ignore Previous Duplicate Marking (<code>--force</code>) ===

By default the deduper will throw an error and stop if a read is already marked as duplicate. The <code>--force</code> option will removes any previous duplicate marking and marks the reads from scratch. The resulting output file will only have reads determined by the deduper marked as duplicates.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0xB04

By default skips reads with any of the following flags set:
* unmapped
* secondary alignment
* fails QC checks
* supplementary reads

The Deduper will not work if secondary or supplementary reads are not excluded. It will not properly find the mates since there will be more than 2 reads with the same mate information.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

== Recalibrate (<code>--recab</code>) ==

This option will recalibrate the input file in addition to deduping.

See [[BamUtil: recab]] for recalibration details.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: dedup

2016-03-06T04:15:36Z

Mktrost: /* Parameters */

[[Category:BamUtil|validate]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>dedup</code> function of <code>[[bamUtil]]</code> =
The <code>dedup</code> option of [[bamUtil]] determines duplicates in a coordinate sorted SAM/BAM file. It either marks or removes the lower quality duplicates.

This tool also contains the option to perform recalibration.

NOTE: This tool does not properly work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.

Potential future features:
* Soft clip overlapping reads (for now, use: [[BamUtil: clipOverlap]])

==Handling Duplicates==

The deduper reads all the alignments in a coordinate-sorted SAM/BAM looking for duplicates, failing if the file is not coordinate-sorted.

The deduper assumes that duplicates in the input BAM file are not marked. When the deduper detects a marked duplicate in the input BAM file, it will throw an error and stop. To override this behavior, use the [[#Ignore Previous Duplicate Marking (--force)|<code>--force</code>]] option; in this mode, alignments that are marked as duplicates in the input file are unmarked before the deduper begins its detection algorithm. The result is that only duplicates detected by the deduper will be marked in or removed from the output file.

The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate. If a mate is not found at the expected position, an error message is printed (once per file) indicating this error. Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads. Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate. If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.

With the default settings this tool should produce similar results as Picard.

Use the [[#Treat Reads with Mates On Different Chromosomes As Single-Ended (--oneChrom)|<code>--oneChrom</code>]] option to treat reads with a mate on a different chromosome as single-ended. This option is useful if you are running the deduper on just a single chromosome. The code will use less memory with this option if mates are found on different chromosomes. (Picard does not specially handle mates on different chromosomes, so the --oneChrom option may produce different results than Picard.)

=== Implementation Notes ===
Duplicates are determined by checking for matching keys.

The Key is comprised of:
# Chromosome
# Orientation (forward/reverse)
# Unclipped Start(forward)/End(reverse)
# Library

Rules:
* Skip Unmapped Reads, they are not marked as duplicate
* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:
*# A paired-end record has the same key (even if the pair is not proper/the mate is unmapped/the mate is not found) -OR-
*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])
* Mark both Paired-End Reads Duplicate if:
# Another paired-end pair has the same set of keys and has a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])

This code assumes that at most 1000 bases are clipped at the start of a read.

==Handling Recalibration==
See [[BamUtil: recab]] for recalibration details.

Recalibration parameters can be applied to deduping when --recab is specified.

== How to use it ==

When <code>dedup</code> is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input SAM/BAM file is required, [[#input File (--in)|input File (--in)]], and must be sorted by coordinate.

The output SAM/BAM file is also required, [[#output File (--out)|output File (--out)]].

Recommended usage with Recalibrator:

/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40

Recommended usage without Recalibrator:

/usr/cluster/bin/bam dedup --in ${INPUT}.bam --out ${OUTPUT}.bam --force --oneChrom

= Usage =
./bam dedup --in <InputBamFile> --out <OutputBamFile> [--minQual <minPhred>] [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--excludeFlags <flag>] [--verbose] [--noeof] [--params] [--recab]

Additional Recalibration Usage is documented at [[BamUtil: recab#Usage|BamUtil: recab -> Usage]]

= Parameters =
<pre>
Required parameters :
--in <infile> : Input BAM file name (must be sorted)
--out <outfile> : Output BAM file name (same order with original file)
Optional parameters :
--minQual <int> : Only add scores over this phred quality when determining a read's quality (default: 15)
--log <logfile> : Log and summary statistics (default: [outfile].log, or stderr if --out starts with '-')
--oneChrom : Treat reads with mates on different chromosomes as single-ended.
--rmDups : Remove duplicates (default is to mark duplicates)
--force : Allow an already mark-duplicated BAM file, unmarking any previously marked
duplicates and apply this duplicate marking logic. Default is to throw errors
and exit when trying to run on an already mark-duplicated BAM
--excludeFlags <flag> : exclude reads with any of these flags set when determining or marking duplicates
by default (0xB04): exclude unmapped, secondary reads, QC failures, and supplementary reads
--verbose : Turn on verbose mode
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings
--recab : Recalibrate in addition to deduping
</pre>
{{PhoneHomeParamDesc}}

Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters|BamUtil: recab -> Parameters]]

== Required Parameters ==
{{inBAMInputFile}}

Note: The input file must be sorted by coordinate.
{{outBAMOutputFile}}

== Optional Parameters==
=== Minimum Quality for Quality Calculations (<code>--minQual</code>) ===

When duplicate reads are encountered, the read with the highest quality is kept.

To determine the quality of a read, all of the phred base quality scores above the <code>--minQual</code> value are added together. If <code>--minQual</code> is not specified, it is defaulted to 15.

=== Output log & Summary Statistics FileName (<code>--log</code>) ===

Output file name for writing logs & summary statistics.

If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

=== Treat Reads with Mates On Different Chromosomes As Single-Ended (<code>--oneChrom</code>) ===

If a read's mate is not found it will not be used for duplicate marking. If you are running on a single chromosome, all read's whose mates are on different chromosomes will not be used for duplicate marking. The <code>--oneChrom</code> option will treat reads with mates on a different chromosome as single-ended.

=== Remove Duplicates (<code>--rmDups</code>) ===

Instead of marking a read as duplicate in the flag, the <code>--rmDups</code> option will remove it from the output BAM file.

=== Ignore Previous Duplicate Marking (<code>--force</code>) ===

By default the deduper will throw an error and stop if a read is already marked as duplicate. The <code>--force</code> option will removes any previous duplicate marking and marks the reads from scratch. The resulting output file will only have reads determined by the deduper marked as duplicates.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0xB04

By default skips reads with any of the following flags set:
* unmapped
* secondary alignment
* fails QC checks
* supplementary reads

This parameter was added in version 1.0.10.

=== Turn on Verbose Mode (<code>--verbose</code>) ===

Turn on verbose logging to get more log messages in the log and to stderr.

{{noeofBGZFParameter}}
{{paramsParameter}}

== Recalibrate (<code>--recab</code>) ==

This option will recalibrate the input file in addition to deduping.

See [[BamUtil: recab]] for recalibration details.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: clipOverlap

2016-03-06T04:01:37Z

Mktrost: /* ASSUMPTIONS/RESTRICTIONS */

[[Category:BamUtil|clipOverlap]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>clipOverlap</code> function of <code>bamUtil</code> =
The <code>clipOverlap</code> option on the [[bamUtil]] executable clips overlapping read pairs.

The input file and resulting output file are sorted by coordinate (or readName if specified in the options).

When a read is clipped from the front:
* the read start position is updated to reflect the clipping.
* the mate's mate start position is updated to reflect the record's new position.
* the record is placed in the output file in the correct location based on the updated position.

To handle coordinate-sorted files, SAM/BAM records are buffered up until it is known that all following records will have a later start position. To prevent the program from running away with memory, a limit is set to the number of records that can be buffered, see [[#Set the SAM/BAMs record buffer size (--poolSize)|<code>--poolSize</code>]] for more information.

When two mates overlap, this tool will clip the record's whose clipped region would have the lowest average quality.

It also checks strand. If a forward strand extends past the end of a reverse strand, that will be clipped. Similarly, if a reverse strand starts before the forward strand, the region prior to the forward strand will be clipped. If the reverse strand occurs entirely before the forward strand, both strands will be entirely clipped. If the [[#Mark entirely clipped reads as unmapped (--unmapped)|<code>--unmapped</code>]] option is specified, then rather than clipping an entire read, it will be marked as unmapped.

The qualities on the two strands remain unchanged even with clipping.

== ASSUMPTIONS/RESTRICTIONS ==

*Assumes the file is sorted by Coordinate (or ReadName if using <code>--readName</code> option)
*Assumes only 2 reads have matching ReadNames (Supplementary and Secondary reads are ignored/skipped by default so will not cause a problem)
**It matches in pairs, so if there are 3, the first 2 will be matched and compared, but the 3rd won't. If there are 4, the first 2 will be matched and the last 2 will be matched and compared.
*Only mapped reads will be clipped
*Assumes that mate information in records are accurate

= Rules for Clipping =

== Clipping from the front==
The first operation after the softclip will be a Match/Mismatch, meaning that any trailing pads, deletions, insertions, or skips will also be soft clipped.
{|border="1" cellspacing="0" cellpadding="2"
! Clip Location !! How it is handled
|-
|If the clip position falls in a skip/deletion
| Removes the entire skip/deletion
|-
|If the position immediately after the clip is a skip/deletion
| Also removes the skip/deletion
|-
|If the position immediately after the clip is an Insert
| Softclips the insert
|-
|If the position immediately after the clip is a Pad
| Removes the pad
|-
| Clip occurs at the last match/mismatch position of the read (the entire read is clipped)
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs after the read ends
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs before the read starts
| Nothing is clipped. The read is not changed.
|}

== Clipping from the back==

{|border="1" cellspacing="0" cellpadding="2"
! Clip Location !! How it is handled
|-
|If the clip position falls in a skip/deletion
| Removes the entire skip/deletion
|-
|If the position immediately before the clip is a deletion/skip/pad
| Remove the deletion/skip/pad
|-
|If the position immediately before the clip is an insertion
| Leave the insertion, even if it results in a 70M3I27S
|-
|Clip occurs at the first position of the read (the entire read is clipped)
| Entire read is soft clipped, preceding insertions remain, 0-based position is left as the original (not modified)
|-
| Clip occurs before the read starts
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs after the read ends
| Nothing is clipped. The read is not changed.
|}

= Usage =
./bam clipOverlap --in <inputFile> --out <outputFile> [--storeOrig <tag>] [--readName] [--stats] [--overlapsOnly] [--excludeFlags <flag>] [--poolSize <numRecords allowed to allocate>] [--poolSkipOverlap] [--noeof] [--params]

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to clip overlaping read pairs for
--out : the SAM/BAM file to be written
Optional Parameters:
--storeOrig : Store the original cigar in the specified tag.
--readName : Original file is sorted by Read Name instead of coordinate.
--stats : Print some statistics on the overlaps.
--overlapsOnly : Only output overlapping read pairs
--excludeFlags : Skip records with any of the specified flags set, default 0xF0C
--unmapped : Mark records that would be completely clipped as unmapped
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings to stderr
Clipping By Coordinate Optional Parameters:
--poolSize : Maximum number of records the program is allowed to allocate
for clipping on Coordinate sorted files. (Default: 1000000)
--poolSkipClip : Skip clipping reads to free of usable records when the
poolSize is hit. The default action is to just clip the
first read in a pair to free up the record.
</pre>
{{PhoneHomeParamDesc}}

== Required Parameters==
{{inBAMInputFile}}
{{outBAMOutputFile}}

== Optional Parameters ==
=== Store the original cigar string in a tag (<code>--storeOrig</code>) ===

Use <code>--storeOrig</code> followed by the two character TAG to store the original CIGAR.

It will be stored with the specified tag as a "Z" tag type.

=== Work on SAM/BAMs sorted by Read Name instead of by coordinate (<code>--readName</code>) ===

If your file is sorted by read name rather than by coordinate, specify <code>--readName</code>. The resulting file will still be sorted by read name.

=== Print Overlap Statistics (<code>--stats</code>)===
Print some basic overlap statistics to stderr.

Output values
* count of the number of overlapping pairs that are clipped
* average of the number of overlapping reference bases that are clipped
* variance of the number of overlapping reference bases that are clipped
* number of times the forward strand is clipped when read pairs overlap
* number of times the reverse strand is clipped when read pairs overlap
* number of times the orientation causes clipping/additional clipping
** reads that are only clipped due to orientation are not counted in the other stats

==== Example Output ====
<pre>
Overlap Statistics:
Number of overlapping pairs: 14
Average # Reference Bases Overlapped: 18.3571
Variance of Reference Bases overlapped: 39.1703
Number of times the forward strand was clipped: 6
Number of times the reverse strand was clipped: 8
Number of times orientation causes additional clipping: 4
</pre>

=== Print Only Overlaping Reads (<code>--overlapsOnly</code>)===
Only output Read Pairs that overlap. Drop all other records.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0xF0C

By default skips reads with any of the following flags set:
* unmapped
* mate unmapped
* secondary alignment
* fails QC checks
* duplicate
* supplementary

=== Mark entirely clipped reads as unmapped (<code>--unmapped</code>)===
Specify this option if instead of marking reads as entirely clipped, mark them as unmapped.

When marking a read as unmapped, it will:
* Set CIGAR to 0
* Set MapQ to 0
* Clear N/A flag fields:
** Proper pair
** Secondary Alignment
** Supplementary Alignment
* Update the Mate's flag to indicate:
** Mate Unmapped
** Not proper pair

{{noeofBGZFParameter}}
{{paramsParameter}}

==Clipping By Coordinate Optional Parameters==
=== Set the SAM/BAMs record buffer size (<code>--poolSize</code>) ===

To handle coordinate sorted files, SAM/BAM records are buffered until it is known that all following records will have a later start position. To prevent the program from running away with memory, a limit is set to the number of records that can be buffered (defaults to 1000000).

If the poolSize is exhausted, the code will write the earliest record awaiting its overlapping mate and any previous records that are being buffered.

Depending on whether or not <code>--poolSkipClip</code> is set, it will either, clip the end of the read at the position where the mate is supposed to start or it will not clip either read. An error message is written to stderr to indicate that one of these has happened and an unsuccessful return value is returned (2: NO_MORE_RECS).

The resulting file will still be sorted by coordinate.

=== Skip Clipping Coordinate Sorted Files When Out of Records (<code>--poolSkipClip</code>) ===

When clipping coordinate sorted SAM/BAM files, we can run out of buffers available in the pool (<code>--poolSize</code>).

By default when we run out of pooled records, we can no longer read in new records, so instead we release some of the stored records. We do this by dropping the first record that is being held awaiting its mate.

This record can either be:
* Clipped starting at its mate's start position until the end of the read (DEFAULT)
* Left as is with no clipping, leaving the mates mates overlapping (specify <code>--poolSkipClip</code>)

With either option, the resulting file will still be sorted by coordinate.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 for success, non-0 for failure).

Returns SamStatus::NO_MORE_RECS, 2, if it was clipping files sorted by coordinate and it ran out of records in the pool so had to clip based on the <code>--poolSkipClip</code> setting.

= Output =

All status messages are written to stderr.

{|border="1" cellspacing="0" cellpadding="2"
! Situation !! Sorted Type !! Output Message
|-
| Everything ran successfully
| ReadName/Coordinate
|
Completed ClipOverlap Successfully.
|-
| Failed to allocate any records
| ReadName/Coordinate
|
Failed to allocate any records.
Failed to complete ClipOverlap.
|-
| Error writing a record
| ReadName
|
Failed to complete ClipOverlap.
|-
| Expected pair to overlap, but 2nd read was not found in the specified position (may be combined with output for running out of pooled records)
| Coordinate
|
Failed to find expected overlapping mates for XX records.
Completed ClipOverlap Successfully.
|-
| Ran out of pooled Records with <code>--poolSkipClip</code> setting
| Coordinate
|
Due to hitting the max record poolSize, had to skip clipping XX records.
Completed ClipOverlap.
|-
| Ran out of pooled Records without <code>--poolSkipClip</code> setting
| Coordinate
|
Due to hitting the max record poolSize, had to default clip XX records.
Completed ClipOverlap.
|-
| Any other error
| Coordinate
|
Failed to complete ClipOverlap.
|}

== Example Output ==
<pre>
Failed to find expected overlapping mates for 2 records.
Due to hitting the max record poolSize, had to default clip 9 records.
Completed ClipOverlap.
</pre>

BamUtil: clipOverlap

2016-03-06T04:00:24Z

Mktrost: /* Parameters */

[[Category:BamUtil|clipOverlap]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>clipOverlap</code> function of <code>bamUtil</code> =
The <code>clipOverlap</code> option on the [[bamUtil]] executable clips overlapping read pairs.

The input file and resulting output file are sorted by coordinate (or readName if specified in the options).

When a read is clipped from the front:
* the read start position is updated to reflect the clipping.
* the mate's mate start position is updated to reflect the record's new position.
* the record is placed in the output file in the correct location based on the updated position.

To handle coordinate-sorted files, SAM/BAM records are buffered up until it is known that all following records will have a later start position. To prevent the program from running away with memory, a limit is set to the number of records that can be buffered, see [[#Set the SAM/BAMs record buffer size (--poolSize)|<code>--poolSize</code>]] for more information.

When two mates overlap, this tool will clip the record's whose clipped region would have the lowest average quality.

It also checks strand. If a forward strand extends past the end of a reverse strand, that will be clipped. Similarly, if a reverse strand starts before the forward strand, the region prior to the forward strand will be clipped. If the reverse strand occurs entirely before the forward strand, both strands will be entirely clipped. If the [[#Mark entirely clipped reads as unmapped (--unmapped)|<code>--unmapped</code>]] option is specified, then rather than clipping an entire read, it will be marked as unmapped.

The qualities on the two strands remain unchanged even with clipping.

== ASSUMPTIONS/RESTRICTIONS ==

*Assumes the file is sorted by Coordinate (or ReadName if using <code>--readName</code> option)
*Assumes only 2 reads have matching ReadNames
**It matches in pairs, so if there are 3, the first 2 will be matched and compared, but the 3rd won't. If there are 4, the first 2 will be matched and the last 2 will be matched and compared.
*Only mapped reads will be clipped
*Assumes that mate information in records are accurate

= Rules for Clipping =

== Clipping from the front==
The first operation after the softclip will be a Match/Mismatch, meaning that any trailing pads, deletions, insertions, or skips will also be soft clipped.
{|border="1" cellspacing="0" cellpadding="2"
! Clip Location !! How it is handled
|-
|If the clip position falls in a skip/deletion
| Removes the entire skip/deletion
|-
|If the position immediately after the clip is a skip/deletion
| Also removes the skip/deletion
|-
|If the position immediately after the clip is an Insert
| Softclips the insert
|-
|If the position immediately after the clip is a Pad
| Removes the pad
|-
| Clip occurs at the last match/mismatch position of the read (the entire read is clipped)
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs after the read ends
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs before the read starts
| Nothing is clipped. The read is not changed.
|}

== Clipping from the back==

{|border="1" cellspacing="0" cellpadding="2"
! Clip Location !! How it is handled
|-
|If the clip position falls in a skip/deletion
| Removes the entire skip/deletion
|-
|If the position immediately before the clip is a deletion/skip/pad
| Remove the deletion/skip/pad
|-
|If the position immediately before the clip is an insertion
| Leave the insertion, even if it results in a 70M3I27S
|-
|Clip occurs at the first position of the read (the entire read is clipped)
| Entire read is soft clipped, preceding insertions remain, 0-based position is left as the original (not modified)
|-
| Clip occurs before the read starts
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs after the read ends
| Nothing is clipped. The read is not changed.
|}

= Usage =
./bam clipOverlap --in <inputFile> --out <outputFile> [--storeOrig <tag>] [--readName] [--stats] [--overlapsOnly] [--excludeFlags <flag>] [--poolSize <numRecords allowed to allocate>] [--poolSkipOverlap] [--noeof] [--params]

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to clip overlaping read pairs for
--out : the SAM/BAM file to be written
Optional Parameters:
--storeOrig : Store the original cigar in the specified tag.
--readName : Original file is sorted by Read Name instead of coordinate.
--stats : Print some statistics on the overlaps.
--overlapsOnly : Only output overlapping read pairs
--excludeFlags : Skip records with any of the specified flags set, default 0xF0C
--unmapped : Mark records that would be completely clipped as unmapped
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings to stderr
Clipping By Coordinate Optional Parameters:
--poolSize : Maximum number of records the program is allowed to allocate
for clipping on Coordinate sorted files. (Default: 1000000)
--poolSkipClip : Skip clipping reads to free of usable records when the
poolSize is hit. The default action is to just clip the
first read in a pair to free up the record.
</pre>
{{PhoneHomeParamDesc}}

== Required Parameters==
{{inBAMInputFile}}
{{outBAMOutputFile}}

== Optional Parameters ==
=== Store the original cigar string in a tag (<code>--storeOrig</code>) ===

Use <code>--storeOrig</code> followed by the two character TAG to store the original CIGAR.

It will be stored with the specified tag as a "Z" tag type.

=== Work on SAM/BAMs sorted by Read Name instead of by coordinate (<code>--readName</code>) ===

If your file is sorted by read name rather than by coordinate, specify <code>--readName</code>. The resulting file will still be sorted by read name.

=== Print Overlap Statistics (<code>--stats</code>)===
Print some basic overlap statistics to stderr.

Output values
* count of the number of overlapping pairs that are clipped
* average of the number of overlapping reference bases that are clipped
* variance of the number of overlapping reference bases that are clipped
* number of times the forward strand is clipped when read pairs overlap
* number of times the reverse strand is clipped when read pairs overlap
* number of times the orientation causes clipping/additional clipping
** reads that are only clipped due to orientation are not counted in the other stats

==== Example Output ====
<pre>
Overlap Statistics:
Number of overlapping pairs: 14
Average # Reference Bases Overlapped: 18.3571
Variance of Reference Bases overlapped: 39.1703
Number of times the forward strand was clipped: 6
Number of times the reverse strand was clipped: 8
Number of times orientation causes additional clipping: 4
</pre>

=== Print Only Overlaping Reads (<code>--overlapsOnly</code>)===
Only output Read Pairs that overlap. Drop all other records.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0xF0C

By default skips reads with any of the following flags set:
* unmapped
* mate unmapped
* secondary alignment
* fails QC checks
* duplicate
* supplementary

=== Mark entirely clipped reads as unmapped (<code>--unmapped</code>)===
Specify this option if instead of marking reads as entirely clipped, mark them as unmapped.

When marking a read as unmapped, it will:
* Set CIGAR to 0
* Set MapQ to 0
* Clear N/A flag fields:
** Proper pair
** Secondary Alignment
** Supplementary Alignment
* Update the Mate's flag to indicate:
** Mate Unmapped
** Not proper pair

{{noeofBGZFParameter}}
{{paramsParameter}}

==Clipping By Coordinate Optional Parameters==
=== Set the SAM/BAMs record buffer size (<code>--poolSize</code>) ===

To handle coordinate sorted files, SAM/BAM records are buffered until it is known that all following records will have a later start position. To prevent the program from running away with memory, a limit is set to the number of records that can be buffered (defaults to 1000000).

If the poolSize is exhausted, the code will write the earliest record awaiting its overlapping mate and any previous records that are being buffered.

Depending on whether or not <code>--poolSkipClip</code> is set, it will either, clip the end of the read at the position where the mate is supposed to start or it will not clip either read. An error message is written to stderr to indicate that one of these has happened and an unsuccessful return value is returned (2: NO_MORE_RECS).

The resulting file will still be sorted by coordinate.

=== Skip Clipping Coordinate Sorted Files When Out of Records (<code>--poolSkipClip</code>) ===

When clipping coordinate sorted SAM/BAM files, we can run out of buffers available in the pool (<code>--poolSize</code>).

By default when we run out of pooled records, we can no longer read in new records, so instead we release some of the stored records. We do this by dropping the first record that is being held awaiting its mate.

This record can either be:
* Clipped starting at its mate's start position until the end of the read (DEFAULT)
* Left as is with no clipping, leaving the mates mates overlapping (specify <code>--poolSkipClip</code>)

With either option, the resulting file will still be sorted by coordinate.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 for success, non-0 for failure).

Returns SamStatus::NO_MORE_RECS, 2, if it was clipping files sorted by coordinate and it ran out of records in the pool so had to clip based on the <code>--poolSkipClip</code> setting.

= Output =

All status messages are written to stderr.

{|border="1" cellspacing="0" cellpadding="2"
! Situation !! Sorted Type !! Output Message
|-
| Everything ran successfully
| ReadName/Coordinate
|
Completed ClipOverlap Successfully.
|-
| Failed to allocate any records
| ReadName/Coordinate
|
Failed to allocate any records.
Failed to complete ClipOverlap.
|-
| Error writing a record
| ReadName
|
Failed to complete ClipOverlap.
|-
| Expected pair to overlap, but 2nd read was not found in the specified position (may be combined with output for running out of pooled records)
| Coordinate
|
Failed to find expected overlapping mates for XX records.
Completed ClipOverlap Successfully.
|-
| Ran out of pooled Records with <code>--poolSkipClip</code> setting
| Coordinate
|
Due to hitting the max record poolSize, had to skip clipping XX records.
Completed ClipOverlap.
|-
| Ran out of pooled Records without <code>--poolSkipClip</code> setting
| Coordinate
|
Due to hitting the max record poolSize, had to default clip XX records.
Completed ClipOverlap.
|-
| Any other error
| Coordinate
|
Failed to complete ClipOverlap.
|}

== Example Output ==
<pre>
Failed to find expected overlapping mates for 2 records.
Due to hitting the max record poolSize, had to default clip 9 records.
Completed ClipOverlap.
</pre>

BamUtil: clipOverlap

2016-03-06T03:59:46Z

Mktrost: /* Skip Records with any of the Specified Flags (--excludeFlags) */

[[Category:BamUtil|clipOverlap]]
[[Category:BAM Software]]
[[Category:Software]]

= Overview of the <code>clipOverlap</code> function of <code>bamUtil</code> =
The <code>clipOverlap</code> option on the [[bamUtil]] executable clips overlapping read pairs.

The input file and resulting output file are sorted by coordinate (or readName if specified in the options).

When a read is clipped from the front:
* the read start position is updated to reflect the clipping.
* the mate's mate start position is updated to reflect the record's new position.
* the record is placed in the output file in the correct location based on the updated position.

To handle coordinate-sorted files, SAM/BAM records are buffered up until it is known that all following records will have a later start position. To prevent the program from running away with memory, a limit is set to the number of records that can be buffered, see [[#Set the SAM/BAMs record buffer size (--poolSize)|<code>--poolSize</code>]] for more information.

When two mates overlap, this tool will clip the record's whose clipped region would have the lowest average quality.

It also checks strand. If a forward strand extends past the end of a reverse strand, that will be clipped. Similarly, if a reverse strand starts before the forward strand, the region prior to the forward strand will be clipped. If the reverse strand occurs entirely before the forward strand, both strands will be entirely clipped. If the [[#Mark entirely clipped reads as unmapped (--unmapped)|<code>--unmapped</code>]] option is specified, then rather than clipping an entire read, it will be marked as unmapped.

The qualities on the two strands remain unchanged even with clipping.

== ASSUMPTIONS/RESTRICTIONS ==

*Assumes the file is sorted by Coordinate (or ReadName if using <code>--readName</code> option)
*Assumes only 2 reads have matching ReadNames
**It matches in pairs, so if there are 3, the first 2 will be matched and compared, but the 3rd won't. If there are 4, the first 2 will be matched and the last 2 will be matched and compared.
*Only mapped reads will be clipped
*Assumes that mate information in records are accurate

= Rules for Clipping =

== Clipping from the front==
The first operation after the softclip will be a Match/Mismatch, meaning that any trailing pads, deletions, insertions, or skips will also be soft clipped.
{|border="1" cellspacing="0" cellpadding="2"
! Clip Location !! How it is handled
|-
|If the clip position falls in a skip/deletion
| Removes the entire skip/deletion
|-
|If the position immediately after the clip is a skip/deletion
| Also removes the skip/deletion
|-
|If the position immediately after the clip is an Insert
| Softclips the insert
|-
|If the position immediately after the clip is a Pad
| Removes the pad
|-
| Clip occurs at the last match/mismatch position of the read (the entire read is clipped)
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs after the read ends
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs before the read starts
| Nothing is clipped. The read is not changed.
|}

== Clipping from the back==

{|border="1" cellspacing="0" cellpadding="2"
! Clip Location !! How it is handled
|-
|If the clip position falls in a skip/deletion
| Removes the entire skip/deletion
|-
|If the position immediately before the clip is a deletion/skip/pad
| Remove the deletion/skip/pad
|-
|If the position immediately before the clip is an insertion
| Leave the insertion, even if it results in a 70M3I27S
|-
|Clip occurs at the first position of the read (the entire read is clipped)
| Entire read is soft clipped, preceding insertions remain, 0-based position is left as the original (not modified)
|-
| Clip occurs before the read starts
| Entire read is soft clipped, 0-based position is left as the original (not modified)
|-
| Clip occurs after the read ends
| Nothing is clipped. The read is not changed.
|}

= Usage =
./bam clipOverlap --in <inputFile> --out <outputFile> [--storeOrig <tag>] [--readName] [--stats] [--overlapsOnly] [--excludeFlags <flag>] [--poolSize <numRecords allowed to allocate>] [--poolSkipOverlap] [--noeof] [--params]

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to clip overlaping read pairs for
--out : the SAM/BAM file to be written
Optional Parameters:
--storeOrig : Store the original cigar in the specified tag.
--readName : Original file is sorted by Read Name instead of coordinate.
--stats : Print some statistics on the overlaps.
--overlapsOnly : Only output overlapping read pairs
--excludeFlags : Skip records with any of the specified flags set, default 0x70C
--unmapped : Mark records that would be completely clipped as unmapped
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings to stderr
Clipping By Coordinate Optional Parameters:
--poolSize : Maximum number of records the program is allowed to allocate
for clipping on Coordinate sorted files. (Default: 1000000)
--poolSkipClip : Skip clipping reads to free of usable records when the
poolSize is hit. The default action is to just clip the
first read in a pair to free up the record.
</pre>
{{PhoneHomeParamDesc}}

== Required Parameters==
{{inBAMInputFile}}
{{outBAMOutputFile}}

== Optional Parameters ==
=== Store the original cigar string in a tag (<code>--storeOrig</code>) ===

Use <code>--storeOrig</code> followed by the two character TAG to store the original CIGAR.

It will be stored with the specified tag as a "Z" tag type.

=== Work on SAM/BAMs sorted by Read Name instead of by coordinate (<code>--readName</code>) ===

If your file is sorted by read name rather than by coordinate, specify <code>--readName</code>. The resulting file will still be sorted by read name.

=== Print Overlap Statistics (<code>--stats</code>)===
Print some basic overlap statistics to stderr.

Output values
* count of the number of overlapping pairs that are clipped
* average of the number of overlapping reference bases that are clipped
* variance of the number of overlapping reference bases that are clipped
* number of times the forward strand is clipped when read pairs overlap
* number of times the reverse strand is clipped when read pairs overlap
* number of times the orientation causes clipping/additional clipping
** reads that are only clipped due to orientation are not counted in the other stats

==== Example Output ====
<pre>
Overlap Statistics:
Number of overlapping pairs: 14
Average # Reference Bases Overlapped: 18.3571
Variance of Reference Bases overlapped: 39.1703
Number of times the forward strand was clipped: 6
Number of times the reverse strand was clipped: 8
Number of times orientation causes additional clipping: 4
</pre>

=== Print Only Overlaping Reads (<code>--overlapsOnly</code>)===
Only output Read Pairs that overlap. Drop all other records.

=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
Skip records with any of the specified flags set, default 0xF0C

By default skips reads with any of the following flags set:
* unmapped
* mate unmapped
* secondary alignment
* fails QC checks
* duplicate
* supplementary

=== Mark entirely clipped reads as unmapped (<code>--unmapped</code>)===
Specify this option if instead of marking reads as entirely clipped, mark them as unmapped.

When marking a read as unmapped, it will:
* Set CIGAR to 0
* Set MapQ to 0
* Clear N/A flag fields:
** Proper pair
** Secondary Alignment
** Supplementary Alignment
* Update the Mate's flag to indicate:
** Mate Unmapped
** Not proper pair

{{noeofBGZFParameter}}
{{paramsParameter}}

==Clipping By Coordinate Optional Parameters==
=== Set the SAM/BAMs record buffer size (<code>--poolSize</code>) ===

To handle coordinate sorted files, SAM/BAM records are buffered until it is known that all following records will have a later start position. To prevent the program from running away with memory, a limit is set to the number of records that can be buffered (defaults to 1000000).

If the poolSize is exhausted, the code will write the earliest record awaiting its overlapping mate and any previous records that are being buffered.

Depending on whether or not <code>--poolSkipClip</code> is set, it will either, clip the end of the read at the position where the mate is supposed to start or it will not clip either read. An error message is written to stderr to indicate that one of these has happened and an unsuccessful return value is returned (2: NO_MORE_RECS).

The resulting file will still be sorted by coordinate.

=== Skip Clipping Coordinate Sorted Files When Out of Records (<code>--poolSkipClip</code>) ===

When clipping coordinate sorted SAM/BAM files, we can run out of buffers available in the pool (<code>--poolSize</code>).

By default when we run out of pooled records, we can no longer read in new records, so instead we release some of the stored records. We do this by dropping the first record that is being held awaiting its mate.

This record can either be:
* Clipped starting at its mate's start position until the end of the read (DEFAULT)
* Left as is with no clipping, leaving the mates mates overlapping (specify <code>--poolSkipClip</code>)

With either option, the resulting file will still be sorted by coordinate.

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 for success, non-0 for failure).

Returns SamStatus::NO_MORE_RECS, 2, if it was clipping files sorted by coordinate and it ran out of records in the pool so had to clip based on the <code>--poolSkipClip</code> setting.

= Output =

All status messages are written to stderr.

{|border="1" cellspacing="0" cellpadding="2"
! Situation !! Sorted Type !! Output Message
|-
| Everything ran successfully
| ReadName/Coordinate
|
Completed ClipOverlap Successfully.
|-
| Failed to allocate any records
| ReadName/Coordinate
|
Failed to allocate any records.
Failed to complete ClipOverlap.
|-
| Error writing a record
| ReadName
|
Failed to complete ClipOverlap.
|-
| Expected pair to overlap, but 2nd read was not found in the specified position (may be combined with output for running out of pooled records)
| Coordinate
|
Failed to find expected overlapping mates for XX records.
Completed ClipOverlap Successfully.
|-
| Ran out of pooled Records with <code>--poolSkipClip</code> setting
| Coordinate
|
Due to hitting the max record poolSize, had to skip clipping XX records.
Completed ClipOverlap.
|-
| Ran out of pooled Records without <code>--poolSkipClip</code> setting
| Coordinate
|
Due to hitting the max record poolSize, had to default clip XX records.
Completed ClipOverlap.
|-
| Any other error
| Coordinate
|
Failed to complete ClipOverlap.
|}

== Example Output ==
<pre>
Failed to find expected overlapping mates for 2 records.
Due to hitting the max record poolSize, had to default clip 9 records.
Completed ClipOverlap.
</pre>

BamUtil: bam2FastQ

2016-03-06T03:53:50Z

Mktrost: /* Parameters */

= Overview of the <code>bam2FastQ</code> function of <code>[[bamUtil]]</code> =
The <code>bam2FastQ</code> option on the [[bamUtil]] converts a BAM file into FastQ files. This is necessary when only BAM files are delivered but a new alignment is desired. By converting BAM to FastQ files new alignments can be done using FastQ files

'''NOTE: Secondary and Supplementary reads are skipped when converting to FastQ. It assumes that there will only be 2 reads (the 2 primary mates) with the same read name that are not secondary or supplementary.'''

'''NOTE: Use the --splitRG option to split reads into read group specific FASTQs.'''

== How to use it ==

When bam2FastQ is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input BAM file is required, [[#input File (--in)|input File (--in)]].

It works on both read/query name and coordinate sorted SAM/BAM files.

If you want to convert a SAM/BAM that is read/query name sorted but the SO field of the header does not specify "queryname", then use the [[#BAM File Is Sorted By Read Name (--readname)|--readName]] option.

When processing files sorted by read name, the only requirement is that matching read names are next to each other. It does not need to be in strict alphabetical order.

Read Names in paired-end FASTQ files are appended with "/1" for the first in the pair, and "/2" for the second in the pair. Override these defaults using [[#First in Pair FastQ ReadName Extension (--firstRNExt)|--firstRNExt]] and [[#Second in Pair FastQ ReadName Extension (--secondRNExt)|--secondRNExt]]

Sequences marked as Reverse strands in the SAM/BAM file are reverse complemented prior to writing to the FASTQ files. To skip this step, specify [[#Do Not Reverse Complement Reverse Strands (--noReverseComp)|--noReverseComp]]

Any errors and a summary of how many pairs and unpaired reads were processed are written to stderr.

'''NOTE: This tool does not work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.'''

'''NOTE: This tool does not split reads into read group specific FASTQs. If you want Read Group specific FASTQ files, first run [[BamUtil: splitBam]] to first split the BAM into 1 BAM per Read Group. Then run bam2FastQ on each bam.'''

=== Output Files ===
By default, this program produces 3 output fastq files.
# unpaired reads
# first end of paired reads
# second end of paired reads

If the [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]] option is specified, the program produces 2 output fastq files.
# unpaired reads
# interleaved paired-end reads

The default fastq file names are determined by taking the base name of the input file and adding an extension for each filetype.
{|border="1" cellspacing="0" cellpadding="2"
! colspan="2"|Default !!colspan="2"|[[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]]
|-
! Output File Contents !! Extension !! Output File Contents !! Extension
|-
|unpaired reads
| .fastq
|unpaired reads
| .fastq
|-
|first end of paired reads
| _1.fastq
| rowspan="2"|interleaved paired-end reads
(both first & second end)
| rowspan="2"|_interleaved.fastq
|-
|second end of paired reads
| _2.fastq
|}

If the inputFile was "myPath/myFile.bam", the resulting fastqs would be:
#myPath/myFile.fastq
#myPath/myFile_1.fastq
#myPath/myFile_2.fastq

With the [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]] option, the resulting fastqs would be:
#myPath/myFile.fastq
#myPath/myFile_interleaved.fastq

Instead of using the inputFile base name as the output file base, you can specify a different base name by using the [[#Output FastQ File Base Name (--outBase)|--outBase]] option.

You can optionally directly specify the output fastq filenames using:
* --firstOut firstReadInAPair.fastq (also used for the interleaved filename with [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]])
* --secondOut secondReadInAPair.fastq
* --unpairedOut unpairedReads.fastq
If any of these are not specified, the <code>--outBase</code> or default is used for that file.

= Usage =
./bam bam2FastQ --in <inputFile> [--readName] [--splitRG] [--qualField <tag>] [--refFile <referenceFile>] [--outBase <outputFileBase>] [--firstOut <1stReadInPairOutFile>] [--merge|--secondOut <2ndReadInPairOutFile>] [--unpairedOut <unpairedOutFile>] [--firstRNExt <firstInPairReadNameEx t>] [--secondRNExt <secondInPairReadNameExt>] [--rnPlus] [--noReverseComp] [--region <chr>[:<pos>[:<base>]]] [--gzip] [--noeof] [--params]

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to convert to FastQ
Optional Parameters:
--readname : Process the BAM as readName sorted instead
of coordinate if the header does not indicate a sort order.
--splitRG : Split into RG specific fastqs.
--qualField : Use the base quality from the specified tag
rather than from the Quality field (default)
--merge : Generate 1 interleaved (merged) FASTQ for paired-ends (unpaired in a separate file)
use firstOut to override the filename of the interleaved file.
--refFile : Reference file for converting '=' in the sequence to the actual base
if '=' are found and the refFile is not specified, 'N' is written to the FASTQ
--firstRNExt : read name extension to use for first read in a pair
default is "/1"
--secondRNExt : read name extension to use for second read in a pair
default is "/2"
--rnPlus : Add the Read Name/extension to the '+' line of the fastq records
--noReverseComp : Do not reverse complement reads marked as reverse
--region : Only convert reads containing the specified region/nucleotide.
Position formatted as: chr:pos:base
pos (0-based) & base are optional.
--gzip : Compress the output FASTQ files using gzip
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings to stderr
Optional OutputFile Names:
--outBase : Base output name for generated output files
--firstOut : Output name for the first in pair file
over-rides setting of outBase
--secondOut : Output name for the second in pair file
over-rides setting of outBase
--unpairedOut : Output name for unpaired reads
over-rides setting of outBase
</pre>

== Required Parameters ==
{{inBAMInputFile}}

== Optional Parameters ==
=== BAM File Is Sorted By Read Name (<code>--readname</code>) ===

The bam2FastQ program by default checks the sort order in the SAM/BAM header when converting to FASTQ, and if that is not specified, assumes it is sorted by coordinate.

To override the default and force it to assume the file is sorted by readname, specify the <code>--readName</code> option

The file does not need to be strictly sorted by read name. The only requirement is that matching read names are next to each other.

=== Split into RG Specific FASTQs (<code>--splitRG</code>) ===

Create RG specific FASTQ files.

Cannot be specified with firstOut/secondOut/unpairedOut since there will be a different filename for each RG.

Cannot write to stdout when <code>--splitRG</code> is specified.

Output filenames will be <outBase>.<RG>_1.fastq, <outBase>.<RG>_2.fastq, and <outBase>.<RG>.fastq. A fastq list file <outBase>.list will be created containing MERGE_NAME (the RG tag's SM value or outBase if the value is empty), fastq 1, fastq 2 (or . if it is a single ended fastq), and the RG tag string.

=== Use the Base Quality from the Specified Tag (<code>--qualField</code>) ===

By default, the quality field is used for the Base Qualities in the FASTQ file. Specify <code>--qualField <tagName></code> to use the base qualities from the specified tag instead of the quality field.

=== Generate 1 Paired-End Output File (<code>--merge</code>) ===

Use the <code>--merge</code> option to generate 1 interleaved (merged) FASTQ for paired-ends instead of 2 files. Unpaired reads are still written to a separate file.

The default extension for the output file is "_interleaved".

Use [[#Output FastQ File Name For the First End of Paired End (--firstOut)|<code>--firstOut</code>]] to override the filename of the interleaved file.

This parameter was added in version 1.0.10.

=== Reference File for Converting '=' in the Sequence to Bases (<code>--refFile</code>) ===
If the SAM/BAM file contains '=' in the sequence instead of the actual bases, the bam2FastQ program needs to convert the '=' back to the bases. To do that it needs the reference. Specify the reference by using <code>--refFile</code> followed by the reference filename.

For example:
./bam bam2FastQ --in myFile.bam --refFile myPath/myRefFile.fa

=== First in Pair FastQ ReadName Extension (<code>--firstRNExt</code>) ===

<code>--firstRNExt</code> overrides the default "/1" that is appended to the Read Name of the first-end of a read pair with the specified value.

=== Second in Pair FastQ ReadName Extension (<code>--secondRNExt</code>) ===

<code>--secondRNExt</code> overrides the default "/2" that is appended to the Read Name of the second-end of a read pair with the specified value.

=== Include the Read Name on the "+" line of the FASTQ (<code>--rnPlus</code>) ===

By default the read name is not included on the "+" line of the FASTQ files. To include the read name and the extension for paired-end reads, specify <code>--rnPlus</code>.

=== Do Not Reverse Complement Reverse Strands (<code>--noReverseComp</code>) ===

By default, reads marked as reverse in the BAM file are reverse complemented prior to writing to the FASTQ files. <code>--noReverseComp</code> disables this feature, and skips the reverse complement step.

For example, if a sequence is ACCGTG marked as reverse, the default FASTQ record will be written as: CACGGT

Specifying <code>--noReverseComp</code> would result in a FASTQ sequence of ACCGTG

=== Only Convert the Specified Region (<code>--region</code>) ===

Only convert reads containing the specified region/nucleotide.

Position formatted as: chr:pos:base

pos (0-based) & base are optional.

{{noeofBGZFParameter}}
{{paramsParameter}}

== Optional Output Filenames ==

=== Output FastQ File Base Name (<code>--outBase</code>) ===

You can replace the default output base name by using the <code>--outBase</code> option.
If the outBase was "myNewPath/myFastQBase", the resulting fastq's would be:
#myNewPath/myFastQBase.fastq
#myNewPath/myFastQBase_1.fastq
#myNewPath/myFastQBase_2.fastq

The value specified by this parameter is overridden by <code>--firstOut</code>, <code>--secondOut</code>, and <code>--unpairedOut</code>, but is used for whichever output files are not specified.

With the [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]] option, the resulting fastq's would instead be:
#myNewPath/myFastQBase.fastq
#myNewPath/myFastQBase_interleaved.fastq

=== Output FastQ File Name For the First End of Paired End (<code>--firstOut</code>) ===

This setting overides the default and <code>--outBase</code> file name.

The entire filename and extension must be specified.

Does not affect the filenames for the second end or for unpaired reads.

For example:
./bam bam2FastQ --in myFile.bam --firstOut myFileEnd1.fastq

=== Output FastQ File Name For the Second End of Paired End (<code>--secondOut</code>) ===

This setting overides the default and <code>--outBase</code> file name.

The entire filename and extension must be specified.

Does not affect the filenames for the first end or for unpaired reads.

For example:
./bam bam2FastQ --in myFile.bam --secondOut myFileEnd2.fastq

=== Output FastQ File Name For Unpaired Reads (<code>--unpairedOut</code>) ===

This setting overides the default and <code>--outBase</code> file names.

The entire filename and extension must be specified.

Does not affect the filenames for the paired-end fastq files.

For example:
./bam bam2FastQ --in myFile.bam --unpairedOut myFileUnpaired.fastq

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: bam2FastQ

2016-03-06T03:36:19Z

Mktrost: /* Usage */

= Overview of the <code>bam2FastQ</code> function of <code>[[bamUtil]]</code> =
The <code>bam2FastQ</code> option on the [[bamUtil]] converts a BAM file into FastQ files. This is necessary when only BAM files are delivered but a new alignment is desired. By converting BAM to FastQ files new alignments can be done using FastQ files

'''NOTE: Secondary and Supplementary reads are skipped when converting to FastQ. It assumes that there will only be 2 reads (the 2 primary mates) with the same read name that are not secondary or supplementary.'''

'''NOTE: Use the --splitRG option to split reads into read group specific FASTQs.'''

== How to use it ==

When bam2FastQ is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input BAM file is required, [[#input File (--in)|input File (--in)]].

It works on both read/query name and coordinate sorted SAM/BAM files.

If you want to convert a SAM/BAM that is read/query name sorted but the SO field of the header does not specify "queryname", then use the [[#BAM File Is Sorted By Read Name (--readname)|--readName]] option.

When processing files sorted by read name, the only requirement is that matching read names are next to each other. It does not need to be in strict alphabetical order.

Read Names in paired-end FASTQ files are appended with "/1" for the first in the pair, and "/2" for the second in the pair. Override these defaults using [[#First in Pair FastQ ReadName Extension (--firstRNExt)|--firstRNExt]] and [[#Second in Pair FastQ ReadName Extension (--secondRNExt)|--secondRNExt]]

Sequences marked as Reverse strands in the SAM/BAM file are reverse complemented prior to writing to the FASTQ files. To skip this step, specify [[#Do Not Reverse Complement Reverse Strands (--noReverseComp)|--noReverseComp]]

Any errors and a summary of how many pairs and unpaired reads were processed are written to stderr.

'''NOTE: This tool does not work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.'''

'''NOTE: This tool does not split reads into read group specific FASTQs. If you want Read Group specific FASTQ files, first run [[BamUtil: splitBam]] to first split the BAM into 1 BAM per Read Group. Then run bam2FastQ on each bam.'''

=== Output Files ===
By default, this program produces 3 output fastq files.
# unpaired reads
# first end of paired reads
# second end of paired reads

If the [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]] option is specified, the program produces 2 output fastq files.
# unpaired reads
# interleaved paired-end reads

The default fastq file names are determined by taking the base name of the input file and adding an extension for each filetype.
{|border="1" cellspacing="0" cellpadding="2"
! colspan="2"|Default !!colspan="2"|[[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]]
|-
! Output File Contents !! Extension !! Output File Contents !! Extension
|-
|unpaired reads
| .fastq
|unpaired reads
| .fastq
|-
|first end of paired reads
| _1.fastq
| rowspan="2"|interleaved paired-end reads
(both first & second end)
| rowspan="2"|_interleaved.fastq
|-
|second end of paired reads
| _2.fastq
|}

If the inputFile was "myPath/myFile.bam", the resulting fastqs would be:
#myPath/myFile.fastq
#myPath/myFile_1.fastq
#myPath/myFile_2.fastq

With the [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]] option, the resulting fastqs would be:
#myPath/myFile.fastq
#myPath/myFile_interleaved.fastq

Instead of using the inputFile base name as the output file base, you can specify a different base name by using the [[#Output FastQ File Base Name (--outBase)|--outBase]] option.

You can optionally directly specify the output fastq filenames using:
* --firstOut firstReadInAPair.fastq (also used for the interleaved filename with [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]])
* --secondOut secondReadInAPair.fastq
* --unpairedOut unpairedReads.fastq
If any of these are not specified, the <code>--outBase</code> or default is used for that file.

= Usage =
./bam bam2FastQ --in <inputFile> [--readName] [--splitRG] [--qualField <tag>] [--refFile <referenceFile>] [--outBase <outputFileBase>] [--firstOut <1stReadInPairOutFile>] [--merge|--secondOut <2ndReadInPairOutFile>] [--unpairedOut <unpairedOutFile>] [--firstRNExt <firstInPairReadNameEx t>] [--secondRNExt <secondInPairReadNameExt>] [--rnPlus] [--noReverseComp] [--region <chr>[:<pos>[:<base>]]] [--gzip] [--noeof] [--params]

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to convert to FastQ
Optional Parameters:
--readname : Process the BAM as readName sorted instead
of coordinate if the header does not indicate a sort order.
--merge : Generate 1 interleaved (merged) FASTQ for paired-ends (unpaired in a separate file)
use firstOut to override the filename of the interleaved file.
--refFile : Reference file for converting '=' in the sequence to the actual base
if '=' are found and the refFile is not specified, 'N' is written to the FASTQ
--firstRNExt : read name extension to use for first read in a pair
default is "/1"
--secondRNExt : read name extension to use for second read in a pair
default is "/2"
--rnPlus : Add the Read Name/extension to the '+' line of the fastq records
--noReverseComp : Do not reverse complement reads marked as reverse
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings to stderr
Optional OutputFile Names:
--outBase : Base output name for generated output files
--firstOut : Output name for the first in pair file
over-rides setting of outBase
--secondOut : Output name for the second in pair file
over-rides setting of outBase
--unpairedOut : Output name for unpaired reads
over-rides setting of outBase
</pre>

== Required Parameters ==
{{inBAMInputFile}}

== Optional Parameters ==
=== BAM File Is Sorted By Read Name (<code>--readname</code>) ===

The bam2FastQ program by default checks the sort order in the SAM/BAM header when converting to FASTQ, and if that is not specified, assumes it is sorted by coordinate.

To override the default and force it to assume the file is sorted by readname, specify the <code>--readName</code> option

The file does not need to be strictly sorted by read name. The only requirement is that matching read names are next to each other.

=== Generate 1 Paired-End Output File (<code>--merge</code>) ===

Use the <code>--merge</code> option to generate 1 interleaved (merged) FASTQ for paired-ends instead of 2 files. Unpaired reads are still written to a separate file.

The default extension for the output file is "_interleaved".

Use [[#Output FastQ File Name For the First End of Paired End (--firstOut)|<code>--firstOut</code>]] to override the filename of the interleaved file.

This parameter was added in version 1.0.10.

=== Reference File for Converting '=' in the Sequence to Bases (<code>--refFile</code>) ===
If the SAM/BAM file contains '=' in the sequence instead of the actual bases, the bam2FastQ program needs to convert the '=' back to the bases. To do that it needs the reference. Specify the reference by using <code>--refFile</code> followed by the reference filename.

For example:
./bam bam2FastQ --in myFile.bam --refFile myPath/myRefFile.fa

=== First in Pair FastQ ReadName Extension (<code>--firstRNExt</code>) ===

<code>--firstRNExt</code> overrides the default "/1" that is appended to the Read Name of the first-end of a read pair with the specified value.

=== Second in Pair FastQ ReadName Extension (<code>--secondRNExt</code>) ===

<code>--secondRNExt</code> overrides the default "/2" that is appended to the Read Name of the second-end of a read pair with the specified value.

=== Include the Read Name on the "+" line of the FASTQ (<code>--rnPlus</code>) ===

By default the read name is not included on the "+" line of the FASTQ files. To include the read name and the extension for paired-end reads, specify <code>--rnPlus</code>.

=== Do Not Reverse Complement Reverse Strands (<code>--noReverseComp</code>) ===

By default, reads marked as reverse in the BAM file are reverse complemented prior to writing to the FASTQ files. <code>--noReverseComp</code> disables this feature, and skips the reverse complement step.

For example, if a sequence is ACCGTG marked as reverse, the default FASTQ record will be written as: CACGGT

Specifying <code>--noReverseComp</code> would result in a FASTQ sequence of ACCGTG

{{noeofBGZFParameter}}
{{paramsParameter}}

== Optional Output Filenames ==

=== Output FastQ File Base Name (<code>--outBase</code>) ===

You can replace the default output base name by using the <code>--outBase</code> option.
If the outBase was "myNewPath/myFastQBase", the resulting fastq's would be:
#myNewPath/myFastQBase.fastq
#myNewPath/myFastQBase_1.fastq
#myNewPath/myFastQBase_2.fastq

The value specified by this parameter is overridden by <code>--firstOut</code>, <code>--secondOut</code>, and <code>--unpairedOut</code>, but is used for whichever output files are not specified.

With the [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]] option, the resulting fastq's would instead be:
#myNewPath/myFastQBase.fastq
#myNewPath/myFastQBase_interleaved.fastq

=== Output FastQ File Name For the First End of Paired End (<code>--firstOut</code>) ===

This setting overides the default and <code>--outBase</code> file name.

The entire filename and extension must be specified.

Does not affect the filenames for the second end or for unpaired reads.

For example:
./bam bam2FastQ --in myFile.bam --firstOut myFileEnd1.fastq

=== Output FastQ File Name For the Second End of Paired End (<code>--secondOut</code>) ===

This setting overides the default and <code>--outBase</code> file name.

The entire filename and extension must be specified.

Does not affect the filenames for the first end or for unpaired reads.

For example:
./bam bam2FastQ --in myFile.bam --secondOut myFileEnd2.fastq

=== Output FastQ File Name For Unpaired Reads (<code>--unpairedOut</code>) ===

This setting overides the default and <code>--outBase</code> file names.

The entire filename and extension must be specified.

Does not affect the filenames for the paired-end fastq files.

For example:
./bam bam2FastQ --in myFile.bam --unpairedOut myFileUnpaired.fastq

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

BamUtil: bam2FastQ

2016-03-06T03:35:04Z

Mktrost: /* Overview of the bam2FastQ function of bamUtil */

= Overview of the <code>bam2FastQ</code> function of <code>[[bamUtil]]</code> =
The <code>bam2FastQ</code> option on the [[bamUtil]] converts a BAM file into FastQ files. This is necessary when only BAM files are delivered but a new alignment is desired. By converting BAM to FastQ files new alignments can be done using FastQ files

'''NOTE: Secondary and Supplementary reads are skipped when converting to FastQ. It assumes that there will only be 2 reads (the 2 primary mates) with the same read name that are not secondary or supplementary.'''

'''NOTE: Use the --splitRG option to split reads into read group specific FASTQs.'''

== How to use it ==

When bam2FastQ is invoked without any arguments the usage information is displayed as described below under [[#Usage|Usage]].

The input BAM file is required, [[#input File (--in)|input File (--in)]].

It works on both read/query name and coordinate sorted SAM/BAM files.

If you want to convert a SAM/BAM that is read/query name sorted but the SO field of the header does not specify "queryname", then use the [[#BAM File Is Sorted By Read Name (--readname)|--readName]] option.

When processing files sorted by read name, the only requirement is that matching read names are next to each other. It does not need to be in strict alphabetical order.

Read Names in paired-end FASTQ files are appended with "/1" for the first in the pair, and "/2" for the second in the pair. Override these defaults using [[#First in Pair FastQ ReadName Extension (--firstRNExt)|--firstRNExt]] and [[#Second in Pair FastQ ReadName Extension (--secondRNExt)|--secondRNExt]]

Sequences marked as Reverse strands in the SAM/BAM file are reverse complemented prior to writing to the FASTQ files. To skip this step, specify [[#Do Not Reverse Complement Reverse Strands (--noReverseComp)|--noReverseComp]]

Any errors and a summary of how many pairs and unpaired reads were processed are written to stderr.

'''NOTE: This tool does not work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.'''

'''NOTE: This tool does not split reads into read group specific FASTQs. If you want Read Group specific FASTQ files, first run [[BamUtil: splitBam]] to first split the BAM into 1 BAM per Read Group. Then run bam2FastQ on each bam.'''

=== Output Files ===
By default, this program produces 3 output fastq files.
# unpaired reads
# first end of paired reads
# second end of paired reads

If the [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]] option is specified, the program produces 2 output fastq files.
# unpaired reads
# interleaved paired-end reads

The default fastq file names are determined by taking the base name of the input file and adding an extension for each filetype.
{|border="1" cellspacing="0" cellpadding="2"
! colspan="2"|Default !!colspan="2"|[[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]]
|-
! Output File Contents !! Extension !! Output File Contents !! Extension
|-
|unpaired reads
| .fastq
|unpaired reads
| .fastq
|-
|first end of paired reads
| _1.fastq
| rowspan="2"|interleaved paired-end reads
(both first & second end)
| rowspan="2"|_interleaved.fastq
|-
|second end of paired reads
| _2.fastq
|}

If the inputFile was "myPath/myFile.bam", the resulting fastqs would be:
#myPath/myFile.fastq
#myPath/myFile_1.fastq
#myPath/myFile_2.fastq

With the [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]] option, the resulting fastqs would be:
#myPath/myFile.fastq
#myPath/myFile_interleaved.fastq

Instead of using the inputFile base name as the output file base, you can specify a different base name by using the [[#Output FastQ File Base Name (--outBase)|--outBase]] option.

You can optionally directly specify the output fastq filenames using:
* --firstOut firstReadInAPair.fastq (also used for the interleaved filename with [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]])
* --secondOut secondReadInAPair.fastq
* --unpairedOut unpairedReads.fastq
If any of these are not specified, the <code>--outBase</code> or default is used for that file.

= Usage =
./bam bam2FastQ --in <inputFile> [--readName] [--refFile <referenceFile>] [--outBase <outputFileBase>] [--firstOut <1stReadInPairOutFile>] [--merge|--secondOut <2ndReadInPairOutFile>] [--unpairedOut <unpairedOutFile>] [--firstRNExt <firstInPairReadNameExt>] [--secondRNExt <secondInPairReadNameExt>] [--rnPlus] [--noReverseComp] [--noeof] [--params]

= Parameters =
<pre>
Required Parameters:
--in : the SAM/BAM file to convert to FastQ
Optional Parameters:
--readname : Process the BAM as readName sorted instead
of coordinate if the header does not indicate a sort order.
--merge : Generate 1 interleaved (merged) FASTQ for paired-ends (unpaired in a separate file)
use firstOut to override the filename of the interleaved file.
--refFile : Reference file for converting '=' in the sequence to the actual base
if '=' are found and the refFile is not specified, 'N' is written to the FASTQ
--firstRNExt : read name extension to use for first read in a pair
default is "/1"
--secondRNExt : read name extension to use for second read in a pair
default is "/2"
--rnPlus : Add the Read Name/extension to the '+' line of the fastq records
--noReverseComp : Do not reverse complement reads marked as reverse
--noeof : Do not expect an EOF block on a bam file.
--params : Print the parameter settings to stderr
Optional OutputFile Names:
--outBase : Base output name for generated output files
--firstOut : Output name for the first in pair file
over-rides setting of outBase
--secondOut : Output name for the second in pair file
over-rides setting of outBase
--unpairedOut : Output name for unpaired reads
over-rides setting of outBase
</pre>

== Required Parameters ==
{{inBAMInputFile}}

== Optional Parameters ==
=== BAM File Is Sorted By Read Name (<code>--readname</code>) ===

The bam2FastQ program by default checks the sort order in the SAM/BAM header when converting to FASTQ, and if that is not specified, assumes it is sorted by coordinate.

To override the default and force it to assume the file is sorted by readname, specify the <code>--readName</code> option

The file does not need to be strictly sorted by read name. The only requirement is that matching read names are next to each other.

=== Generate 1 Paired-End Output File (<code>--merge</code>) ===

Use the <code>--merge</code> option to generate 1 interleaved (merged) FASTQ for paired-ends instead of 2 files. Unpaired reads are still written to a separate file.

The default extension for the output file is "_interleaved".

Use [[#Output FastQ File Name For the First End of Paired End (--firstOut)|<code>--firstOut</code>]] to override the filename of the interleaved file.

This parameter was added in version 1.0.10.

=== Reference File for Converting '=' in the Sequence to Bases (<code>--refFile</code>) ===
If the SAM/BAM file contains '=' in the sequence instead of the actual bases, the bam2FastQ program needs to convert the '=' back to the bases. To do that it needs the reference. Specify the reference by using <code>--refFile</code> followed by the reference filename.

For example:
./bam bam2FastQ --in myFile.bam --refFile myPath/myRefFile.fa

=== First in Pair FastQ ReadName Extension (<code>--firstRNExt</code>) ===

<code>--firstRNExt</code> overrides the default "/1" that is appended to the Read Name of the first-end of a read pair with the specified value.

=== Second in Pair FastQ ReadName Extension (<code>--secondRNExt</code>) ===

<code>--secondRNExt</code> overrides the default "/2" that is appended to the Read Name of the second-end of a read pair with the specified value.

=== Include the Read Name on the "+" line of the FASTQ (<code>--rnPlus</code>) ===

By default the read name is not included on the "+" line of the FASTQ files. To include the read name and the extension for paired-end reads, specify <code>--rnPlus</code>.

=== Do Not Reverse Complement Reverse Strands (<code>--noReverseComp</code>) ===

By default, reads marked as reverse in the BAM file are reverse complemented prior to writing to the FASTQ files. <code>--noReverseComp</code> disables this feature, and skips the reverse complement step.

For example, if a sequence is ACCGTG marked as reverse, the default FASTQ record will be written as: CACGGT

Specifying <code>--noReverseComp</code> would result in a FASTQ sequence of ACCGTG

{{noeofBGZFParameter}}
{{paramsParameter}}

== Optional Output Filenames ==

=== Output FastQ File Base Name (<code>--outBase</code>) ===

You can replace the default output base name by using the <code>--outBase</code> option.
If the outBase was "myNewPath/myFastQBase", the resulting fastq's would be:
#myNewPath/myFastQBase.fastq
#myNewPath/myFastQBase_1.fastq
#myNewPath/myFastQBase_2.fastq

The value specified by this parameter is overridden by <code>--firstOut</code>, <code>--secondOut</code>, and <code>--unpairedOut</code>, but is used for whichever output files are not specified.

With the [[#Generate 1 Paired-End Output File (--merge)|<code>--merge</code>]] option, the resulting fastq's would instead be:
#myNewPath/myFastQBase.fastq
#myNewPath/myFastQBase_interleaved.fastq

=== Output FastQ File Name For the First End of Paired End (<code>--firstOut</code>) ===

This setting overides the default and <code>--outBase</code> file name.

The entire filename and extension must be specified.

Does not affect the filenames for the second end or for unpaired reads.

For example:
./bam bam2FastQ --in myFile.bam --firstOut myFileEnd1.fastq

=== Output FastQ File Name For the Second End of Paired End (<code>--secondOut</code>) ===

This setting overides the default and <code>--outBase</code> file name.

The entire filename and extension must be specified.

Does not affect the filenames for the first end or for unpaired reads.

For example:
./bam bam2FastQ --in myFile.bam --secondOut myFileEnd2.fastq

=== Output FastQ File Name For Unpaired Reads (<code>--unpairedOut</code>) ===

This setting overides the default and <code>--outBase</code> file names.

The entire filename and extension must be specified.

Does not affect the filenames for the paired-end fastq files.

For example:
./bam bam2FastQ --in myFile.bam --unpairedOut myFileUnpaired.fastq

{{PhoneHomeParameters}}

= Return Value =

Returns -1 if input parameters are invalid.

Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

GotCloud: Alignment Pipeline

2016-01-28T22:39:14Z

Mktrost: /* Optional Configurable Settings */

Back to parent: [[GotCloud]]

== Overview of Alignment Pipeline Steps ==
The Alignment/Mapping Pipeline takes [http://en.wikipedia.org/wiki/FASTQ_format FASTQ files] and generates recalibrated [[BAM|BAM (Binary Sequence Alignment/Map format) files]] from them.

[[File:MappingSteps.png]]

== Running the GotCloud Alignment Pipeline ==

The alignment pipeline is run using the <code>align</code> option of the <code>gotcloud</code> script. This option calls <code>align.pl</code> found in the <code>bin/</code> directory under the <code>gotcloud</code> installation.

Use the <code>--conf</code> parameter followed by the configuration file to specify the configuration to use for this run of the alignment pipeline.

You must specify the input list of FASTQs mapped to sample id to tell the alignment pipeline what files to process. You can do this by setting either:
* <code>FASTQ_LIST</code> in the configuration file
* <code>--list</code> on the command-line

You must specify an output directory to tell the alignment pipeline where to write its output by either setting:
* <code>OUT_DIR</code> in the configuration file
* <code>--outdir</code> on the command-line

'''Example of a Basic Alignment Command'''
gotcloud align --conf myAlignTest.conf --outdir ~/gotcloudOutput/align/

===Running the Automated Test===

The automated test runs the alignment pipeline on a small set of test data and checks that the results against expected results validating that GotCloud is installed correctly.

*Run alignment pipeline test:
gotcloud align --test OUTPUT_DIR
where OUTPUT_DIR is the directory where you want to store the test results

If you see "Successfully ran the test case, congratulations!", then you are ready to align samples.

== Input Data:==
*Raw Sequence (FASTQ) files
*FASTQ List file mapping fastq pairs to sample (optional: Read Group information)
*Reference files
*(Optional) Configuration file to override default options

=== Raw Sequence (FASTQ) files ===

These are the FASTQ files that need to be mapped to BAM files.

These files are specified in the [[#FASTQ List File|FASTQ List File]].

=== FASTQ List File ===
This file specifies the FASTQ files that need to be processed. It maps the FASTQ pairs to the associated Sample ID. Optionally Read Group information for the FASTQ pairs can be specified. If the Read Group information is not specified, it is inferred.

This file is specified either via the command line parameter <code>--list</code> or via the configuration file setting <code>FASTQ_LIST</code>.

The command-line setting takes precedence over the configuration file setting.

The FASTQ list is a tab delimited file that starts with a header line. The columns may be in any order.

Following the header line, there is one line per single-end read and one line per paired-end read (only 1 line per pair).

Required Column Names:
* MERGE_NAME - base name for the resulting BAM file for the sample (used to group multiple fastqs or fastq pairs into a single BAM)
** The SAMPLE column can be specified instead of MERGE_NAME. SAMPLE will be used for both the sample and the base name.
* FASTQ1 - name of the fastq or the first in the pair if paired-end. (Only 1 line per pair)

Optional Column Names:
* FASTQ2 - name of the 2nd fastq in paired-end reads. Specify '.' if the column exists, but this line is single-ended.
* RGID - Read Group ID for this entry
** If this field is not specified, the first line of the fastq will be used to determine the RG.
*** If the first line does not match the expected format for determining RG, incrementing numbers per fastq file will be used.
* SAMPLE - Sample Name for this entry
** If SAMPLE is not specified, MERGE_NAME will be used for the sample name
* LIBRARY - Library for this entry
** If LIBRARY is not specified, the sample name will be used
* CENTER - Center Name for this entry
** If CENTER is not specified, it will default to "unknown"
* PLATFORM - Platform for this entry
** If PLATFORM is not specified, it will default to ILLUMINA

The RGID, SAMPLE, LIBRARY, CENTER, and PLATFORM are used to populate the Read Group information for this entry.

MERGE_NAME FASTQ1 FASTQ2 RGID SAMPLE LIBRARY CENTER PLATFORM
Sample1 fastq/S1/F1_R1.fastq.gz fastq/S1/F1_R2.fastq.gz RGID1 SampleID1 Lib1 UM ILLUMINA
Sample1 fastq/S1/F2_R1.fastq.gz fastq/S1/F2_R2.fastq.gz RGID1a SampleID1 Lib1 UM ILLUMINA
Sample2 fastq/S2/F1_R1.fastq.gz fastq/S2/F1_R2.fastq.gz RGID2 SampleID2 Lib2 UM ILLUMINA
Sample2 fastq/S2/F2.fastq.gz . RGID2 SampleID2 Lib2 UM ILLUMINA

The <code>--fastq_prefix</code>/<code>FASTQ_PREFIX</code> setting can be used to specify a prefix to the FASTQ1/FASTQ2 file paths that should be applied before using the files.

=== Reference Files ===
See [[GotCloud: Genetic Reference and Resource Files]] for detailed information about the multiple required reference files for the alignment pipeline, including:
* How to obtain default references
* Configuration keys & default values
* How to generate your own references
* How to point GotCloud to your reference files

Required Reference File Types:
* [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]]
* [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF Files|DBSNP VCF Files]]
* [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF Files|HapMap3 VCF Files]]

=== Configuration File ===
{{:GotCloud: Configuration}}

==== Recommended Settings ====

As of GotCloud version 1.16, the alignment pipeline uses <code>bwa mem</code> by default. Prior to version 1.16, the default aligner was <code>bwa aln</code>.

You can override the defaults by setting in your configuration file:
* to use <code>bwa mem</code> (you do not need to set this in version 1.16 and later since it is the default)
MAP_TYPE = BWA_MEM
* to use <code>bwa aln</code> (you do not need to set this prior to version 1.16 since it is the default)
MAP_TYPE = BWA

==== Additional Required Settings ====

See [[#FASTQ List File|FASTQ List File]] for how to set the index file either via command line options or via configuration.

==== Turning Off Optional Steps====
Quality Control steps can be disabled.

To Disable QPLOT, remove qplot from the PER_MERGE_STEPS configuration by setting:
PER_MERGE_STEPS = verifyBamID index recab

To Disable VerifyBamID, remove qplot from the PER_MERGE_STEPS configuration by setting:
PER_MERGE_STEPS = qplot index recab

==== Optional Configurable Settings ====
You may want to adjust the amount of memory/threads that are used:

There are additional configurable settings, but these are the ones most likely to be adjusted.

* BWA_THREADS = -t N
** Fill in the N with the number of threads you want BWA to run with, default is 1
* BWA_QUAL = -q N
** Fill in the N with the trim quality you want BWA aln to run with, default is 15. This parameter is only applied to bwa aln. It is not used for BWA_MEM.
* BWA_MEM_OPTS =
** Specify any additional bwa mem options using this parameter.
* SORT_MAX_MEM = 2000000000
** Maximum amount of memory used by samtools sort after running bwa

== Running the Alignment Pipeline ==

=== Command-Line Options ===
* help - print usage
* test OUTPUT_DIR - run the test example placing the output in a user specified OUTPUT_DIR. No other options are required.
* outdir OUTPUT_DIR - directory for the output
** May also be specified via OUT_DIR in the configuration file
** Required to be set either via command-line or configuration
* conf CONFIG_FILE - configuration file
* list FASTQ_LIST_FILE_NAME - name of the fastq list file
** May also be specified via FASTQ_LIST in the configuration file
** Required to be set either via command-line or configuration
* ref_dir REFERENCE_DIR - value to set config key REF_DIR to, overriding other values, REF_DIR can then be used inside config files.
** May also be specified via REF_DIR in the configuration file
* ref_prefix REFERENCE_DIR - path to prepend to non-absolute REF paths.
** May also be specified via REF_PREFIX in the configuration file
* fastq_prefix FASTQ_PATH - prefix path to the fastq files specified in the FASTQ_LIST
** May also be specified via FASTQ_PREFIX in the configuration file
* base_prefix BASE_PATH - prefix path to the prepend to fastq/ref files without absolute paths
** May also be specified via BASE_PREFIX in the configuration file
* keepTmp - Do not remove the temporary files (removed by default)
** May also be specified via KEEP_TMP in the configuration file
* keepLog - Do not remove the intermediate log files (removed by default)
** May also be specified via KEEP_LOG in the configuration file
* numjobs N - Replace N with the number of samples that should be processed in parallel
* threads N - Replace N with the number of targets in each makefile that should be run in parallel
* dryrun - Create the Makefile, but do not run it
* maxlocaljobs N - Replace N with the maximum number of jobs that can be run locally (no batchtype specified). Default is 10.
* batchtype TYPE - Tells GotCloud the specified batch type to send jobs to the client nodes
** May also be specified via BATCH_TYPE in the configuration file
** Can be: mosix, slurm, slurmi, pbs, sge, sgei
* batchopts OPTS - Tells GotCloud the options to pass onto the batch system
** May also be specified via BATCH_OPTS in the configuration file
* noPhoneHome - disable the phone home logic
* gotcloudroot DIR - Specifies an alternate path to other gotcloud files rather than using the path to the gotcloud/align.pl.
Note: Command-line options take priority over configuration file settings

===Running the Alignment Pipeline===
Run <code>gotcloud align</code> with the appropriate command-line parameters.

Example:
gotcloud align --conf config.txt --outdir output

This step generates 1 Makefile per sample in the output/Makefiles/ directory and then automatically runs them. The Makefiles contain all of the information to run each sample.

If you only want to generate the makefiles and not run them, use the <code>--dryrun</code> option. It will generate the Makefiles and print instructions for running the Makefiles.

Each Makefile is independent and can be run in parallel and across a cloud.

On success, you will see:
Processing finished in nn secs with no errors reported

If processing fails part way through, you can pick up where you left off by rerunning gotcloud or the make command.

=== Alignment Pipeline Output ===
Upon successful completion of the alignment pipeline, you should see the following files/ subdirectories under the user specified output directory:
* '''bam.list''' - file containing sample->BAM mapping that can be used in other GotCloud pipelines
* '''bams/''' - contains the final BAM and bai (BAM index) files
** '''*.recal.bam'''
** '''*.recal.bam.bai'''
** ''*.recal.bam.bai.done'' - temp file indicating this step completed successfully
** ''*.recal.bam.done'' - temp file indicating this step completed successfully
** *.recal.bam.metrics - dedup & recalibration log
** *.recal.bam.qemp - recalibration tables
* Makefiles/ - contains the Makefiles and logs used by GotCloud to run the alignment pipeline
* '''QCFiles/''' - contains quality control results if quality control is not disabled
** VerifyBamID Output - see [[VerifyBamID#A_guideline_to_interpret_output_files|VerifyBamID: A guideline to interpret output files]] for more information
*** *.genoCheck.depthRG - depth distribution of the sequence reads per read group
*** *.genoCheck.depthSM - depth distribution of the sequence reads per sample
*** ''*.genoCheck.done'' - temp file indicating this step completed successfully
*** *.genoCheck.selfRG - per-readGroup statistics describing how well each lane matches to the annotated sample
*** '''*.genoCheck.selfSM''' - main output file containing the contamination estimate; per-sample statistics describing how well the sample matches to the annotated sample
**** Check the 'FREEMIX' column for genotype-free estimate of contamination 0-1 scale, the lower, the better
**** If [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, possible contamination
** Qplot Output - see: [[QPLOT#Diagnose_sequencing_quality|QPLOT: Diagnose sequencing quality]] for more info on how to use QPLOT results
*** ''*.qplot.done'' - temp file indicating this step completed successfully
*** '''*.qplot.R''' - Rscript that can be used to generate the pdf graphs
*** '''*.qplot.stats''' - sample statistics
* tmp/ - contains intermediate files (most are deleted unless --keepTmp is specified)
* *.OK - one OK file per sample; indicates the Sample successfully completed alignment

You should also see a <code>.OK</code> for each Sample in the index file.

If you do not see these <code>.OK</code> files, then your Alignment Pipeline failed.

'''On success, the bams/ directory contains the final BAMs and bais.'''

Releasing GotCloud

2015-10-29T03:00:38Z

Mktrost:

Back to parent: [[GotCloud]]

Software for GotCloud is kept in a '''git''' repository on '''github'''.

We also create a Debian package and tar file for GotCloud.

To release a new version of GotCloud:
# I recommend creating a new directory:
#: <code> cd ~/dev</code>
#: <code> git clone git://github.com/statgen/gotcloud.git</code>
#: <code> cd gotcloud</code>
# Build the source
#: <code> cd src</code>
#: <code> make</code>
# Run the tests, ensure they all succeed
#: <code>./gotcloud align --test testDir</code>
#: <code>./gotcloud snpcall --test testDir</code>
#: <code>./gotcloud ldrefine --test testDir</code>
#: <code>./gotcloud indel --test testDir</code>
# Update the version number (if you have not already).
#* Edit release_version.txt
#* Commit the new release number
#*: <code>git add release_version.txt</code>
#*: <code>git commit -m "Release 1.XX"</code>
#* Push to github
#*: <code>git push</code>
# Checkout latest branch and merge master & push
#: <code>git checkout latest</code>
#: <code>git merge master</code>
#: <code>git checkout master</code>
#: <code>git push</code>
# Create a release tag
#: <code>git tag -a gotcloud.1.XX -m 'version 1.XX'</code>
# Push tag
#: <code> git push origin gotcloud.1.XX</code>
# Create the debian packages
#* Generate the packages
#*: <code> debian/makedeb.sh bin =</code>
#*: <code> debian/makedeb.sh test =</code>
# Generate the release notes on github
#: On github, login
#::Goto Tags, Select "Draft New Release"
#::* Select the "Tag version" you created
#::* Fill in the release notes
# Update wiki release documentation (for full release)
#* http://genome.sph.umich.edu/wiki/GotCloud:_Versions
# Send Email to google group (for full release)
#* gotcloud@googlegroups.com
#* I usually do it from the groups site as an announcement & post on behalf of GotCloud (I may need to modify your permissions to do this)
#** https://groups.google.com/forum/?hl=en#!forum/gotcloud
# Install on mario
#* In the gotcloud directory where you generated the deb files, run:
#*: <code>debian/install_gotcloud_without_root.sh 1.16 /net/mario</code>
#* Install the latest gotcloud in the gotcloud.old directory (named by version) so someone can run a specific version
#*: <code>debian/install_gotcloud_without_root.sh 1.16 /net/mario/gotcloud.old</code>
#* Rename to the version name:
#*: <code>mv /net/mario/gotcloud.old/gotcloud /net/mario/gotcloud.old/gotcloud.1.XX</code>
# Send email to csg group to inform them of the newly installed version on the cluster and of any relevant changes
#* gotcloudUsers@umich.edu

User talk:Pjvh

2015-08-19T19:00:59Z

Mktrost: Welcome!

User:Pjvh

2015-08-19T19:00:58Z

Mktrost: Creating user page for new user.

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

GotCloud: Creating a New Pipeline

2015-06-30T18:56:30Z

Mktrost: /* Example Pipelines Created */

== Creating a New BAM Processing Pipeline ==

GotCloud allows you to configure new basic BAM processing pipelines via configuration.

To define new processing pipelines, you will use Configuration sections to define both the pipeline and each of the steps. So first you need to understand how configuration sections work.

=== GotCloud Configuration Sections ===

GotCloud configuration files can be broken into sections:
* Section names are specified between square brakets (<code>[]</code>)
*: <pre>[sectionName]</pre>
** Any configuration settings specified after the section header belong to that section
** A section can be specified multiple times in the file and the configuration settings are accumulated
** To access a value for a key defined in another section, use <code>$(otherSectionName/keyName)</code>

* If a section is not specified, the configuration settings belong to the <code>global</code> section
** The <code>global</code> section does not need to be specified at the beginning of the file (it is the default section).
** Additional <code>global</code> settings can be set later in the file after other settings, by defining the explicitly section:
**: <pre>[global]</pre>

* Sections can be derived from another section
** All sections automatically derive from <code>[global]</code>
** A derived section inherits all the configuration settings from its parent sections
*** Parent settings are overridden by redefining the configuration key/value pair
** A parent section is specified following a semicolon <code>:</code> on the section definition line:
**: <pre>[childSectionName] : parentSectionName</pre>

<ul>
<li> Section specific configuration settings are specified on the lines following the section definition:
<dd><pre>[section1]
KEY1 = VAL1
KEY2 = VAL2

[section2]
KEY1 = VAL1_2
KEY3 = VAL3</pre></dd>
</li>
</ul>
=== Defining a New Pipeline ===

There are 2 parts for creating a new pipeline
# [[#Overall Pipeline Definition|Overall Pipeline Definition]]
#* Basics for the overall pipeline
#* '''NOTE: Currently, configurations set in the overall pipeline's section do not by default pass onto the step's configurations'''
# [[#Configure Each Step|Configure Each Step]]

==== Overall Pipeline Definition ====
<ol>
<li> Define a new configuration section for your pipeline
<ul><li> Example:</li></ul>
<dd> <pre>[pipelineName]</pre>
</li>
<li>Define the steps in this pipeline using the key <code>STEPS</code> under that section
<ul><li> Example:</li></ul>
<dd><pre>[pipelineName]
STEPS = stepName1 stepName2 stepName3</pre>
<ul><li> Note: each step must have its own configuration section</li></ul>
</li>
</ol>

Optional Overall Pipeline Settings:
* BATCH_OPTS
* BATCH_TYPE
* IGNORE_SM_CHECK - turn off the default validation that the @RG SM tag matches the bam list sample name.
* IGNORE_REF_CHR_CHECK - turn off the default validation that checks that all of the BAM's chromosomes are in the reference file - eventually we may update to just validate those in CHRS.
* OUT_DIR
* BAM_LIST
* REF
* REF_FAI
* MULTIPLE_TARGET_MAP
* UNIFORM_TARGET_BED
* OFFSET_OFF_TARET
* CHRS - defines which chromosomes to run.
* UNIT_CHUNK
* NO_CRAM - do not allow CRAM files as input
* MAKE_BASE_NAME_PIPE - base makefile name
* MAKE_OPTS - otpions to pass to the make command that runs the jobs.
* BAM_DEPEND - set to TRUE if you want the BAM file to be included as a make dependency

NOTES:
* The BAM_LIST file can contain config values within it - the overall pipeline section will be checked for those config values.
* By default if a value is not defined in the section, it will check global.

==== Configure Each Step ====
'''Create a section for each step'''
* Example: <code>[stepName1]</code>

====Required keys for each step:====

# <code>DEPEND</code> - dependencies for this step
#: Valid Values (separate multiple dependencies with a space):
#:*<code>BAM</code>
#:*Name of step that must complete prior to this step
#:*PER_SAMPLE_BAM??? can only be BAM or PER_SAMPLE_BAM
#<code>OUTPUT</code> - name of output file
#* See below for temporary keys for step iteration
#<code>CMD</code> - command for running the step
#* See below for temporary keys for step iteration

====Optional Step Settings:====
General Settings:
* <code>LOCAL</code> - run the step locally rather than on the cluster
* <code>NEED_BAI</code> - Set if a step requires a BAI file
** Per chromosome steps always require a BAI file
** Tells GotCloud to fail if a BAI can't be found
* <code>BAM_DEPEND</code> - Add the BAM file as a Makefile dependency for this step

Settings to limit which samples this step runs on:
* <code>SAMPLES</code> - use this to define a step to run only for samples with a single BAM or multiple BAMs (merging)
*: Possible values:
*:* <code>MULTI_BAM</code> - run the step only for samples that have multiple BAMs
*:* <code>SINGLE_BAM</code> - run the step only for samples that have one BAM
*Deprecated settings - still in pipeline.pl and may or may not work:
** <code>MULTI_ONLY</code> - set to non-blank if step should run if there are more than 1 input per output.
** <code>SINGLE_ONLY</code> - set to non-blank if step should run if there is only 1 input per output.

Joining multiple inputs for a single output:
* Can occur if there are multiple dependencies
* Can occur if a step runs at a more generic iteration level than a dependency
* <code>INPUT_JOIN</code> - value to pass to perl "join" command for joining multiple inputs for each output.
** Looks across all dependencies
* <code>dependStepName_JOIN</code> - how to join the "dependStepName"'s output into the command line for a step that depends on it if there are multiple outputs per input of this step
** Substitutes <code>?(${depend}/OUTPUT)</code> with perl "join" using the specified value to join multiple outputs for that dependency

Log Output filenames
* <code>FILELIST</code> - writes/appends the iteration's output file name into the specified file list.
** Typically will be used in a later "merge" step
** See below for temporary keys for step iteration that can be used in this filename
*** Temporary keys can be more general than those in OUTPUT, but cannot be more specific.

====Iterating a command for each Bam/Sample/Chromosome/Region====
Temporary keys are used when iterating a command per BAM/sample/chromosome/region.
* Specify using <code>?()</code> rather than <code>$()</code>
* Temporary keys can be used in:
** <code>OUTPUT</code>
** <code>CMD</code>
** <code>FILELIST</code>
* They will be substituted as it iterates
* How to iterate a command is determined by the temporary keys in <code>OUTPUT</code>
* Temporary Keys for determining iterations:
** <code>?(BAM)</code> - per BAM per sample
** <code>?(SAMPLE)</code> - per sample
** <code>?(CHR)</code> - per chromosome
** <code>?(START)</code> - Per region of a Chromosome (must also include <code>?(CHR)</code>):
* Additional Temporary Keys:
** <code>?(END)</code> - end of the region - only used if <code>?(START)</code> is also specified.
** <code>?(INPUT)</code>
** <code>?(${depend}/OUTPUT)</code>

'''Notes:'''
* Currently each step iteration will:
** be its own Makefile target/.OK file
** run independently on the cluster

== Command Line Parameters ==
Required Parameters:
* <code>--name</code> <pipelineName> - name of the pipeline to run
* <code>--conf</code> <configuration file> - configuration file to use

NOTE: Currently, any "overrides" are for the global setting only - not for the pipeline/step.
* this needs to be fixed so they can override the pipeline settings

Optional Parameters:
* <code>--ignoreSmCheck</code> - overrides <code>IGNORE_SM_CHECK</code>
* <code>--ignoreRefChrCheck</code> - overrides <code>IGNORE_REF_CHR_CHECK</code>
* <code>--verbose</code> <number> - verbose value passed to the loadConf method

Optional Parameters like SnpCall:
* <code>--numjobs|numjobs</code> <number> - number of jobs to run in parallel
* <code>--maxlocaljobs</code> <number> - number of jobs to allow to run when batchtype is local (default 10) - does not validate for commands running LOCAL
* <code>--region</code> <region to process> - like snpcall, specifies a single region to process
* <code>--bam_list|list|bamlist|bam_index|bamindex</code> <bam list file> - overrides
<code>BAM_LIST</code> list of sample\tbam to process
* <code>--out_dir|outdir</code> <output directory> - overrides <code>OUT_DIR</code>
* <code>--batchtype</code> <type> - overrides <code>BATCHTYPE</code>
* <code>--batchopts</code> <options> - overrides <code>BATCHOPTS</code>
* <code>--chrs|chroms</code> <comma separated chromosomes> - overrides <code>CHRS</code> (CHRS is space separated - commas are converted to spaces)
* <code>--ref_dir|refdir</code> <reference directory> - overrides <code>REF_DIR</code>
* <code>--ref_prefix|refprefix</code> <prefix> - overrides <code>REF_PREFIX</code>
* <code>--bam_prefix|bamprefix</code> <prefix> - overrides <code>BAM_PREFIX</code>
* <code>--base_prefix|baseprefix</code> <prefix> - overrides <code>BASE_PREFIX</code>
* <code>--gotcloudroot|gcroot</code> <path to gotcloud> - by default gotcloud root is determined from the path to the pipeline script, but this setting overrides that.
* <code>--help</code> - print Usage
* <code>--test</code> <test directory> - run the test code (just for indel right now)

Unused command line options:
* In the code, but are not actually used:
* <code>--keeptmp</code> - overrides <code>KEEP_TMP</code>
* <code>--keeplog</code> - overrides <code>KEEP_LOG</code>

== Example Pipelines Created ==
Look for sections & <code>STEPS</code> in the defaults.
https://github.com/statgen/gotcloud/blob/master/bin/gotcloudDefaults.conf
https://github.com/statgen/gotcloud/blob/alignPrep/bin/gotcloudDefaults.conf

GotCloud: Creating a New Pipeline

2015-06-30T18:51:55Z

Mktrost: /* Overall Pipeline Definition */

GotCloud: Creating a New Pipeline

2015-06-30T18:50:33Z

Mktrost: /* Defining a New Pipeline */

== Creating a New BAM Processing Pipeline ==

GotCloud allows you to configure new basic BAM processing pipelines via configuration.

To define new processing pipelines, you will use Configuration sections to define both the pipeline and each of the steps. So first you need to understand how configuration sections work.

=== GotCloud Configuration Sections ===

GotCloud configuration files can be broken into sections:
* Section names are specified between square brakets (<code>[]</code>)
*: <pre>[sectionName]</pre>
** Any configuration settings specified after the section header belong to that section
** A section can be specified multiple times in the file and the configuration settings are accumulated
** To access a value for a key defined in another section, use <code>$(otherSectionName/keyName)</code>

* If a section is not specified, the configuration settings belong to the <code>global</code> section
** The <code>global</code> section does not need to be specified at the beginning of the file (it is the default section).
** Additional <code>global</code> settings can be set later in the file after other settings, by defining the explicitly section:
**: <pre>[global]</pre>

* Sections can be derived from another section
** All sections automatically derive from <code>[global]</code>
** A derived section inherits all the configuration settings from its parent sections
*** Parent settings are overridden by redefining the configuration key/value pair
** A parent section is specified following a semicolon <code>:</code> on the section definition line:
**: <pre>[childSectionName] : parentSectionName</pre>

<ul>
<li> Section specific configuration settings are specified on the lines following the section definition:
<dd><pre>[section1]
KEY1 = VAL1
KEY2 = VAL2

[section2]
KEY1 = VAL1_2
KEY3 = VAL3</pre></dd>
</li>
</ul>
=== Defining a New Pipeline ===

There are 2 parts for creating a new pipeline
# [[#Overall Pipeline Definition|Overall Pipeline Definition]]
#* Basics for the overall pipeline
#* '''NOTE: Currently, configurations set in the overall pipeline's section do not by default pass onto the step's configurations'''
# [[#Configure Each Step|Configure Each Step]]

==== Overall Pipeline Definition ====
<ol>
<li> Define a new configuration section for your pipeline
<ul><li> Example:</li></ul>
<dd> <pre>[pipelineName]</pre>
</li>
<li>Define the steps in this pipeline using the key <code>STEPS</code> under that section
<ul><li> Example:</li></ul>
<dd><pre>[pipelineName]
STEPS = stepName1 stepName2 stepName3</pre>
<ul><li> Note: each step must have its own configuration section</li></ul>
</li>
</ol>

Optional Overall Pipeline Settings:
* BATCH_OPTS
* BATCH_TYPE
* IGNORE_SM_CHECK - turn off the default validation that the @RG SM tag matches the bam list sample name.
* IGNORE_REF_CHR_CHECK
* OUT_DIR
* BAM_LIST
* REF
* REF_FAI
* MULTIPLE_TARGET_MAP
* UNIFORM_TARGET_BED
* OFFSET_OFF_TARET
* CHRS - defines which chromosomes to run.
* UNIT_CHUNK
* NO_CRAM - do not allow CRAM files as input
* MAKE_BASE_NAME_PIPE - base makefile name
* MAKE_OPTS - otpions to pass to the make command that runs the jobs.
* BAM_DEPEND - set to TRUE if you want the BAM file to be included as a make dependency

NOTES:
* The BAM_LIST file can contain config values within it - the overall pipeline section will be checked for those config values.
* By default if a value is not defined in the section, it will check global.

==== Configure Each Step ====
'''Create a section for each step'''
* Example: <code>[stepName1]</code>

====Required keys for each step:====

# <code>DEPEND</code> - dependencies for this step
#: Valid Values (separate multiple dependencies with a space):
#:*<code>BAM</code>
#:*Name of step that must complete prior to this step
#:*PER_SAMPLE_BAM??? can only be BAM or PER_SAMPLE_BAM
#<code>OUTPUT</code> - name of output file
#* See below for temporary keys for step iteration
#<code>CMD</code> - command for running the step
#* See below for temporary keys for step iteration

====Optional Step Settings:====
General Settings:
* <code>LOCAL</code> - run the step locally rather than on the cluster
* <code>NEED_BAI</code> - Set if a step requires a BAI file
** Per chromosome steps always require a BAI file
** Tells GotCloud to fail if a BAI can't be found
* <code>BAM_DEPEND</code> - Add the BAM file as a Makefile dependency for this step

Settings to limit which samples this step runs on:
* <code>SAMPLES</code> - use this to define a step to run only for samples with a single BAM or multiple BAMs (merging)
*: Possible values:
*:* <code>MULTI_BAM</code> - run the step only for samples that have multiple BAMs
*:* <code>SINGLE_BAM</code> - run the step only for samples that have one BAM
*Deprecated settings - still in pipeline.pl and may or may not work:
** <code>MULTI_ONLY</code> - set to non-blank if step should run if there are more than 1 input per output.
** <code>SINGLE_ONLY</code> - set to non-blank if step should run if there is only 1 input per output.

Joining multiple inputs for a single output:
* Can occur if there are multiple dependencies
* Can occur if a step runs at a more generic iteration level than a dependency
* <code>INPUT_JOIN</code> - value to pass to perl "join" command for joining multiple inputs for each output.
** Looks across all dependencies
* <code>dependStepName_JOIN</code> - how to join the "dependStepName"'s output into the command line for a step that depends on it if there are multiple outputs per input of this step
** Substitutes <code>?(${depend}/OUTPUT)</code> with perl "join" using the specified value to join multiple outputs for that dependency

Log Output filenames
* <code>FILELIST</code> - writes/appends the iteration's output file name into the specified file list.
** Typically will be used in a later "merge" step
** See below for temporary keys for step iteration that can be used in this filename
*** Temporary keys can be more general than those in OUTPUT, but cannot be more specific.

====Iterating a command for each Bam/Sample/Chromosome/Region====
Temporary keys are used when iterating a command per BAM/sample/chromosome/region.
* Specify using <code>?()</code> rather than <code>$()</code>
* Temporary keys can be used in:
** <code>OUTPUT</code>
** <code>CMD</code>
** <code>FILELIST</code>
* They will be substituted as it iterates
* How to iterate a command is determined by the temporary keys in <code>OUTPUT</code>
* Temporary Keys for determining iterations:
** <code>?(BAM)</code> - per BAM per sample
** <code>?(SAMPLE)</code> - per sample
** <code>?(CHR)</code> - per chromosome
** <code>?(START)</code> - Per region of a Chromosome (must also include <code>?(CHR)</code>):
* Additional Temporary Keys:
** <code>?(END)</code> - end of the region - only used if <code>?(START)</code> is also specified.
** <code>?(INPUT)</code>
** <code>?(${depend}/OUTPUT)</code>

'''Notes:'''
* Currently each step iteration will:
** be its own Makefile target/.OK file
** run independently on the cluster

== Command Line Parameters ==
Required Parameters:
* <code>--name</code> <pipelineName> - name of the pipeline to run
* <code>--conf</code> <configuration file> - configuration file to use

NOTE: Currently, any "overrides" are for the global setting only - not for the pipeline/step.
* this needs to be fixed so they can override the pipeline settings

Optional Parameters:
* <code>--ignoreSmCheck</code> - overrides <code>IGNORE_SM_CHECK</code>
* <code>--ignoreRefChrCheck</code> - overrides <code>IGNORE_REF_CHR_CHECK</code>
* <code>--verbose</code> <number> - verbose value passed to the loadConf method

Optional Parameters like SnpCall:
* <code>--numjobs|numjobs</code> <number> - number of jobs to run in parallel
* <code>--maxlocaljobs</code> <number> - number of jobs to allow to run when batchtype is local (default 10) - does not validate for commands running LOCAL
* <code>--region</code> <region to process> - like snpcall, specifies a single region to process
* <code>--bam_list|list|bamlist|bam_index|bamindex</code> <bam list file> - overrides
<code>BAM_LIST</code> list of sample\tbam to process
* <code>--out_dir|outdir</code> <output directory> - overrides <code>OUT_DIR</code>
* <code>--batchtype</code> <type> - overrides <code>BATCHTYPE</code>
* <code>--batchopts</code> <options> - overrides <code>BATCHOPTS</code>
* <code>--chrs|chroms</code> <comma separated chromosomes> - overrides <code>CHRS</code> (CHRS is space separated - commas are converted to spaces)
* <code>--ref_dir|refdir</code> <reference directory> - overrides <code>REF_DIR</code>
* <code>--ref_prefix|refprefix</code> <prefix> - overrides <code>REF_PREFIX</code>
* <code>--bam_prefix|bamprefix</code> <prefix> - overrides <code>BAM_PREFIX</code>
* <code>--base_prefix|baseprefix</code> <prefix> - overrides <code>BASE_PREFIX</code>
* <code>--gotcloudroot|gcroot</code> <path to gotcloud> - by default gotcloud root is determined from the path to the pipeline script, but this setting overrides that.
* <code>--help</code> - print Usage
* <code>--test</code> <test directory> - run the test code (just for indel right now)

Unused command line options:
* In the code, but are not actually used:
* <code>--keeptmp</code> - overrides <code>KEEP_TMP</code>
* <code>--keeplog</code> - overrides <code>KEEP_LOG</code>

== Example Pipelines Created ==
Look for sections & <code>STEPS</code> in the defaults.
https://github.com/statgen/gotcloud/blob/master/bin/gotcloudDefaults.conf

GotCloud: Creating a New Pipeline

2015-06-30T18:50:18Z

Mktrost:

== Creating a New BAM Processing Pipeline ==

GotCloud allows you to configure new basic BAM processing pipelines via configuration.

To define new processing pipelines, you will use Configuration sections to define both the pipeline and each of the steps. So first you need to understand how configuration sections work.

=== GotCloud Configuration Sections ===

GotCloud configuration files can be broken into sections:
* Section names are specified between square brakets (<code>[]</code>)
*: <pre>[sectionName]</pre>
** Any configuration settings specified after the section header belong to that section
** A section can be specified multiple times in the file and the configuration settings are accumulated
** To access a value for a key defined in another section, use <code>$(otherSectionName/keyName)</code>

* If a section is not specified, the configuration settings belong to the <code>global</code> section
** The <code>global</code> section does not need to be specified at the beginning of the file (it is the default section).
** Additional <code>global</code> settings can be set later in the file after other settings, by defining the explicitly section:
**: <pre>[global]</pre>

* Sections can be derived from another section
** All sections automatically derive from <code>[global]</code>
** A derived section inherits all the configuration settings from its parent sections
*** Parent settings are overridden by redefining the configuration key/value pair
** A parent section is specified following a semicolon <code>:</code> on the section definition line:
**: <pre>[childSectionName] : parentSectionName</pre>

<ul>
<li> Section specific configuration settings are specified on the lines following the section definition:
<dd><pre>[section1]
KEY1 = VAL1
KEY2 = VAL2

[section2]
KEY1 = VAL1_2
KEY3 = VAL3</pre></dd>
</li>
</ul>
=== Defining a New Pipeline ===

There are 2 parts for creating a new pipeline
# [[#Overall Pipeline Definition|Overall Pipeline Definition]]
#* Basics for the overall pipeline
#* '''NOTE: Curretnly, configurations set in the overall pipeline's section do not by default pass onto the step's configurations'''
# [[#Configure Each Step|Configure Each Step]]

==== Overall Pipeline Definition ====
<ol>
<li> Define a new configuration section for your pipeline
<ul><li> Example:</li></ul>
<dd> <pre>[pipelineName]</pre>
</li>
<li>Define the steps in this pipeline using the key <code>STEPS</code> under that section
<ul><li> Example:</li></ul>
<dd><pre>[pipelineName]
STEPS = stepName1 stepName2 stepName3</pre>
<ul><li> Note: each step must have its own configuration section</li></ul>
</li>
</ol>

Optional Overall Pipeline Settings:
* BATCH_OPTS
* BATCH_TYPE
* IGNORE_SM_CHECK - turn off the default validation that the @RG SM tag matches the bam list sample name.
* IGNORE_REF_CHR_CHECK
* OUT_DIR
* BAM_LIST
* REF
* REF_FAI
* MULTIPLE_TARGET_MAP
* UNIFORM_TARGET_BED
* OFFSET_OFF_TARET
* CHRS - defines which chromosomes to run.
* UNIT_CHUNK
* NO_CRAM - do not allow CRAM files as input
* MAKE_BASE_NAME_PIPE - base makefile name
* MAKE_OPTS - otpions to pass to the make command that runs the jobs.
* BAM_DEPEND - set to TRUE if you want the BAM file to be included as a make dependency

NOTES:
* The BAM_LIST file can contain config values within it - the overall pipeline section will be checked for those config values.
* By default if a value is not defined in the section, it will check global.

==== Configure Each Step ====
'''Create a section for each step'''
* Example: <code>[stepName1]</code>

====Required keys for each step:====

# <code>DEPEND</code> - dependencies for this step
#: Valid Values (separate multiple dependencies with a space):
#:*<code>BAM</code>
#:*Name of step that must complete prior to this step
#:*PER_SAMPLE_BAM??? can only be BAM or PER_SAMPLE_BAM
#<code>OUTPUT</code> - name of output file
#* See below for temporary keys for step iteration
#<code>CMD</code> - command for running the step
#* See below for temporary keys for step iteration

====Optional Step Settings:====
General Settings:
* <code>LOCAL</code> - run the step locally rather than on the cluster
* <code>NEED_BAI</code> - Set if a step requires a BAI file
** Per chromosome steps always require a BAI file
** Tells GotCloud to fail if a BAI can't be found
* <code>BAM_DEPEND</code> - Add the BAM file as a Makefile dependency for this step

Settings to limit which samples this step runs on:
* <code>SAMPLES</code> - use this to define a step to run only for samples with a single BAM or multiple BAMs (merging)
*: Possible values:
*:* <code>MULTI_BAM</code> - run the step only for samples that have multiple BAMs
*:* <code>SINGLE_BAM</code> - run the step only for samples that have one BAM
*Deprecated settings - still in pipeline.pl and may or may not work:
** <code>MULTI_ONLY</code> - set to non-blank if step should run if there are more than 1 input per output.
** <code>SINGLE_ONLY</code> - set to non-blank if step should run if there is only 1 input per output.

Joining multiple inputs for a single output:
* Can occur if there are multiple dependencies
* Can occur if a step runs at a more generic iteration level than a dependency
* <code>INPUT_JOIN</code> - value to pass to perl "join" command for joining multiple inputs for each output.
** Looks across all dependencies
* <code>dependStepName_JOIN</code> - how to join the "dependStepName"'s output into the command line for a step that depends on it if there are multiple outputs per input of this step
** Substitutes <code>?(${depend}/OUTPUT)</code> with perl "join" using the specified value to join multiple outputs for that dependency

Log Output filenames
* <code>FILELIST</code> - writes/appends the iteration's output file name into the specified file list.
** Typically will be used in a later "merge" step
** See below for temporary keys for step iteration that can be used in this filename
*** Temporary keys can be more general than those in OUTPUT, but cannot be more specific.

====Iterating a command for each Bam/Sample/Chromosome/Region====
Temporary keys are used when iterating a command per BAM/sample/chromosome/region.
* Specify using <code>?()</code> rather than <code>$()</code>
* Temporary keys can be used in:
** <code>OUTPUT</code>
** <code>CMD</code>
** <code>FILELIST</code>
* They will be substituted as it iterates
* How to iterate a command is determined by the temporary keys in <code>OUTPUT</code>
* Temporary Keys for determining iterations:
** <code>?(BAM)</code> - per BAM per sample
** <code>?(SAMPLE)</code> - per sample
** <code>?(CHR)</code> - per chromosome
** <code>?(START)</code> - Per region of a Chromosome (must also include <code>?(CHR)</code>):
* Additional Temporary Keys:
** <code>?(END)</code> - end of the region - only used if <code>?(START)</code> is also specified.
** <code>?(INPUT)</code>
** <code>?(${depend}/OUTPUT)</code>

'''Notes:'''
* Currently each step iteration will:
** be its own Makefile target/.OK file
** run independently on the cluster

== Command Line Parameters ==
Required Parameters:
* <code>--name</code> <pipelineName> - name of the pipeline to run
* <code>--conf</code> <configuration file> - configuration file to use

NOTE: Currently, any "overrides" are for the global setting only - not for the pipeline/step.
* this needs to be fixed so they can override the pipeline settings

Optional Parameters:
* <code>--ignoreSmCheck</code> - overrides <code>IGNORE_SM_CHECK</code>
* <code>--ignoreRefChrCheck</code> - overrides <code>IGNORE_REF_CHR_CHECK</code>
* <code>--verbose</code> <number> - verbose value passed to the loadConf method

Optional Parameters like SnpCall:
* <code>--numjobs|numjobs</code> <number> - number of jobs to run in parallel
* <code>--maxlocaljobs</code> <number> - number of jobs to allow to run when batchtype is local (default 10) - does not validate for commands running LOCAL
* <code>--region</code> <region to process> - like snpcall, specifies a single region to process
* <code>--bam_list|list|bamlist|bam_index|bamindex</code> <bam list file> - overrides
<code>BAM_LIST</code> list of sample\tbam to process
* <code>--out_dir|outdir</code> <output directory> - overrides <code>OUT_DIR</code>
* <code>--batchtype</code> <type> - overrides <code>BATCHTYPE</code>
* <code>--batchopts</code> <options> - overrides <code>BATCHOPTS</code>
* <code>--chrs|chroms</code> <comma separated chromosomes> - overrides <code>CHRS</code> (CHRS is space separated - commas are converted to spaces)
* <code>--ref_dir|refdir</code> <reference directory> - overrides <code>REF_DIR</code>
* <code>--ref_prefix|refprefix</code> <prefix> - overrides <code>REF_PREFIX</code>
* <code>--bam_prefix|bamprefix</code> <prefix> - overrides <code>BAM_PREFIX</code>
* <code>--base_prefix|baseprefix</code> <prefix> - overrides <code>BASE_PREFIX</code>
* <code>--gotcloudroot|gcroot</code> <path to gotcloud> - by default gotcloud root is determined from the path to the pipeline script, but this setting overrides that.
* <code>--help</code> - print Usage
* <code>--test</code> <test directory> - run the test code (just for indel right now)

Unused command line options:
* In the code, but are not actually used:
* <code>--keeptmp</code> - overrides <code>KEEP_TMP</code>
* <code>--keeplog</code> - overrides <code>KEEP_LOG</code>

== Example Pipelines Created ==
Look for sections & <code>STEPS</code> in the defaults.
https://github.com/statgen/gotcloud/blob/master/bin/gotcloudDefaults.conf

GotCloud: Creating a New Pipeline

2015-06-30T18:12:30Z

Mktrost: /* Configure Each Step */

== Creating a New BAM Processing Pipeline ==

GotCloud allows you to configure new basic BAM processing pipelines via configuration.

To define new processing pipelines, you will use Configuration sections to define both the pipeline and each of the steps. So first you need to understand how configuration sections work.

=== GotCloud Configuration Sections ===

GotCloud configuration files can be broken into sections:
* Section names are specified between square brakets (<code>[]</code>)
*: <pre>[sectionName]</pre>
** Any configuration settings specified after the section header belong to that section
** A section can be specified multiple times in the file and the configuration settings are accumulated
** To access a value for a key defined in another section, use <code>$(otherSectionName/keyName)</code>

* If a section is not specified, the configuration settings belong to the <code>global</code> section
** The <code>global</code> section does not need to be specified at the beginning of the file (it is the default section).
** Additional <code>global</code> settings can be set later in the file after other settings, by defining the explicitly section:
**: <pre>[global]</pre>

* Sections can be derived from another section
** All sections automatically derive from <code>[global]</code>
** A derived section inherits all the configuration settings from its parent sections
*** Parent settings are overridden by redefining the configuration key/value pair
** A parent section is specified following a semicolon <code>:</code> on the section definition line:
**: <pre>[childSectionName] : parentSectionName</pre>

<ul>
<li> Section specific configuration settings are specified on the lines following the section definition:
<dd><pre>[section1]
KEY1 = VAL1
KEY2 = VAL2

[section2]
KEY1 = VAL1_2
KEY3 = VAL3</pre></dd>
</li>
</ul>
=== Defining a New Pipeline ===

There are 2 parts for creating a new pipeline
# [[#Overall Pipeline Definition|Overall Pipeline Definition]]
#* Basics for the overall pipeline
#* '''NOTE: Curretnly, configurations set in the overall pipeline's section do not by default pass onto the step's configurations'''
# [[#Configure Each Step|Configure Each Step]]

==== Overall Pipeline Definition ====
<ol>
<li> Define a new configuration section for your pipeline
<ul><li> Example:</li></ul>
<dd> <pre>[pipelineName]</pre>
</li>
<li>Define the steps in this pipeline using the key <code>STEPS</code> under that section
<ul><li> Example:</li></ul>
<dd><pre>[pipelineName]
STEPS = stepName1 stepName2 stepName3</pre>
<ul><li> Note: each step must have its own configuration section</li></ul>
</li>
</ol>

Optional Overall Pipeline Settings:
* BATCH_OPTS
* BATCH_TYPE
* IGNORE_SM_CHECK - turn off the default validation that the @RG SM tag matches the bam list sample name.
* IGNORE_REF_CHR_CHECK
* OUT_DIR
* BAM_LIST
* REF
* REF_FAI
* MULTIPLE_TARGET_MAP
* UNIFORM_TARGET_BED
* OFFSET_OFF_TARET
* CHRS
* UNIT_CHUNK
* NO_CRAM - do not allow CRAM files as input
* MAKE_BASE_NAME_PIPE - base makefile name
* MAKE_OPTS - otpions to pass to the make command that runs the jobs.
* BAM_DEPEND - set to TRUE if you want the BAM file to be included as a make dependency

NOTES:
* The BAM_LIST file can contain config values within it - the overall pipeline section will be checked for those config values.
* By default if a value is not defined in the section, it will check global.

==== Configure Each Step ====
'''Create a section for each step'''
* Example: <code>[stepName1]</code>

====Required keys for each step:====

# <code>DEPEND</code> - dependencies for this step
#: Valid Values (separate multiple dependencies with a space):
#:*<code>BAM</code>
#:*Name of step that must complete prior to this step
#:*PER_SAMPLE_BAM??? can only be BAM or PER_SAMPLE_BAM
#<code>OUTPUT</code> - name of output file
#* See below for temporary keys for step iteration
#<code>CMD</code> - command for running the step
#* See below for temporary keys for step iteration

====Optional Step Settings:====
General Settings:
* <code>LOCAL</code> - run the step locally rather than on the cluster
* <code>NEED_BAI</code> - Set if a step requires a BAI file
** Per chromosome steps always require a BAI file
** Tells GotCloud to fail if a BAI can't be found
* <code>BAM_DEPEND</code> - Add the BAM file as a Makefile dependency for this step

Settings to limit which samples this step runs on:
* <code>SAMPLES</code> - use this to define a step to run only for samples with a single BAM or multiple BAMs (merging)
*: Possible values:
*:* <code>MULTI_BAM</code> - run the step only for samples that have multiple BAMs
*:* <code>SINGLE_BAM</code> - run the step only for samples that have one BAM
*Deprecated settings - still in pipeline.pl and may or may not work:
** <code>MULTI_ONLY</code> - set to non-blank if step should run if there are more than 1 input per output.
** <code>SINGLE_ONLY</code> - set to non-blank if step should run if there is only 1 input per output.

Joining multiple inputs for a single output:
* Can occur if there are multiple dependencies
* Can occur if a step runs at a more generic iteration level than a dependency
* <code>INPUT_JOIN</code> - value to pass to perl "join" command for joining multiple inputs for each output.
** Looks across all dependencies
* <code>dependStepName_JOIN</code> - how to join the "dependStepName"'s output into the command line for a step that depends on it if there are multiple outputs per input of this step
** Substitutes <code>?(${depend}/OUTPUT)</code> with perl "join" using the specified value to join multiple outputs for that dependency

Log Output filenames
* <code>FILELIST</code> - writes/appends the iteration's output file name into the specified file list.
** Typically will be used in a later "merge" step
** See below for temporary keys for step iteration that can be used in this filename
*** Temporary keys can be more general than those in OUTPUT, but cannot be more specific.

====Iterating a command for each Bam/Sample/Chromosome/Region====
Temporary keys are used when iterating a command per BAM/sample/chromosome/region.
* Specify using <code>?()</code> rather than <code>$()</code>
* Temporary keys can be used in:
** <code>OUTPUT</code>
** <code>CMD</code>
** <code>FILELIST</code>
* They will be substituted as it iterates
* How to iterate a command is determined by the temporary keys in <code>OUTPUT</code>
* Temporary Keys for determining iterations:
** <code>?(BAM)</code> - per BAM per sample
** <code>?(SAMPLE)</code> - per sample
** <code>?(CHR)</code> - per chromosome
** <code>?(START)</code> - Per region of a Chromosome (must also include <code>?(CHR)</code>):
* Additional Temporary Keys:
** <code>?(END)</code> - end of the region - only used if <code>?(START)</code> is also specified.
** <code>?(INPUT)</code>
** <code>?(${depend}/OUTPUT)</code>

'''Notes:'''
* Currently each step iteration will:
** be its own Makefile target/.OK file
** run independently on the cluster

GotCloud: Creating a New Pipeline

2015-06-30T17:08:18Z

Mktrost: /* Defining a New Pipeline */

GotCloud: FAQs

2015-06-23T14:54:30Z

Mktrost: /* Variant Calling Pipeline FAQs */

= GotCloud Frequently Asked Questions (FAQs) =

This page hopes to answer some of the FAQs about [[GotCloud]].

If you cannot find the answer to your question here, please [[GotCloud#Getting Help with GotCloud|contact us]].

== General FAQs ==

== Compiling FAQs ==
; How do I compile
: <pre>cd src/; make;</pre>

;bam_tview_curses.c<nowiki>:5:20: error: curses.h: No such file or directory</nowiki> bam_tview_curses.c<nowiki>:7:2: warning: #warning "_CURSES_LIB=1 but NCURSES_VERSION not defined; tview is NOT compiled"</nowiki> bam_tview_curses.c<nowiki>:287:2: warning: #warning "No curses library is available; tview with curses is disabled."</nowiki> make[2]<nowiki>:</nowiki> *** [bam_tview_curses.o] Error 1 make[2]<nowiki>:</nowiki> Leaving directory `/group/onel-lab/GeneticResources/bin/gotcloud/1.16/gotcloud-gotcloud.1.16/src/samtools' make[1]<nowiki>:</nowiki> *** [all-recur] Error 1 make[1]<nowiki>:</nowiki> Leaving directory `/group/onel-lab/GeneticResources/bin/gotcloud/1.16/gotcloud-gotcloud.1.16/src/samtools' make<nowiki>:</nowiki> *** [samtools] Error 2
: This error is due to curses not being installed on your machine. To fix this error, you need to disable curses in samtools & samtools-hybrid
:* Modify: <code>gotcloud/src/samtools/Makefile</code> and <code>gotcloud/src/samtools-hybrid/Makefile</code>
:*# Replace:
:*#: <code>-D_CURSES_LIB=1</code>
:*#: with:
:*#: <code>-D_CURSES_LIB=0</code>
:*# Replace:
:*#: <code>LIBCURSES= -lcurses # -lXCurses</code>
:*#: with:
:*#: <code>LIBCURSES= # -lcurses # -lXCurses</code>

;mkdir -p premo/build; cd premo/build; cmake .. /bin/sh<nowiki>: cmake:</nowiki> command not found make<nowiki>:</nowiki> *** [premo] Error 127
: This occurs if you do not have cmake installed on your machine.
:* Since premo is not used by default, just remove premo from the build
:** premo is only used for the mosaik aligner (non-default) and for MEI calling (which is not yet implemented)
:* Modify: <code>gotcloud/src/Makefile</code>
:** Remove <code>premo</code> from the <code>EXES = </code>... line

; <nowiki>bed.cpp:62: error: call of overloaded ‘to_string(int32_t&)’ is ambiguous</nowiki> <nowiki>/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:2604: note: candidates are: std::string std::to_string(long long int)</nowiki> <nowiki>/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:2610: note: std::string std::to_string(long long unsigned int)</nowiki> <nowiki>/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:2616: note: std::string std::to_string(long double)</nowiki> <nowiki>make[1]: *** [bed.o] Error 1</nowiki> <nowiki>make[1]: Leaving directory `/group/onel-lab/GeneticResources/bin/gotcloud/1.16/gotcloud-gotcloud.1.16/src/vt'</nowiki> <nowiki>make: *** [vt] Error 2</nowiki>
: std::to_string is defined in c++11. However, not all compilers have updated to accept all overloads of to_string.
:* You need to cast the integer to an overload that is defined:
:** Edit line 62 of <code>gotcloud/src/vt/bed.cpp</code>
:**:From:
:**::<code>return this->chrom + ":" + std::to_string(this->start1) + "-" + std::to_string(this->end1);</code>
:**:To:
:**::<code>return this->chrom + ":" + std::to_string(static_cast<long long>(this->start1)) + "-" + std::to_string(static_cast<long long>(this->end1));</code>

;CMake Error<nowiki>:</nowiki> The current CMakeCache.txt directory /path1/gotcloud/src/premo/build/CMakeCache.txt is different than the directory /path2/gotcloud/src/premo/build where CMakeCache.txt was created. This may result in binaries being created in the wrong place. If you are not sure, reedit the CMakeCache.txt <nowiki>CMake Error: The source "/path1/gotcloud/src/premo/CMakeLists.txt" does not match the source "/path2/gotcloud/src/premo/CMakeLists.txt" used to generate cache. Re-run cmake with a different source directory.
</nowiki>
: This compilation error likely occurred because you moved GotCloud to a new directory and premo (one of the tools included in GotCloud) was still pointing to the old location.
:*Remove src/premo/build/CMakeCache.txt and recompile.

== Amazon FAQs ==

== Alignment Pipeline FAQs ==
; Error "read <> mapped to 'chr1' at POS 0 to -1, flag 69,0 has BIN 4680 but should be 266824".         "Fix it by using BAM->SAM->BAM to force a recalculation of the BIN field."         "Fail to index the BAM file"
:This error appears to be due to BWA reference files that do not match the version of BWA used to run GotCloud align. Run the following to regenerate the BWA reference files with the GotCloud version of BWA.
::<pre>gotcloud/bin/bwa index path/to/your/ref.fa</pre>

== Variant Calling Pipeline FAQs ==
; make<nowiki>: *** [outdir/vcfs/chr20/chr20.hardfiltered.vcf.gz.OK] Error 1</nowiki>
: Check the log file at: <code>outdir/vcfs/chr20/chr20.hardfiltered.sites.vcf.summary.log</code>
: If it says:
:: '''[tabix] the index file either does not exist or is older than the vcf file. Please reindex. ERROR: vcf-summary, failed to open ''/path/to/refVcf.vcf.gz'', exit code: 1'''
: It means that your VCF file is newer than the associated index (.tbi) file.
:* If you are using a downloaded set of reference files, and the time stamps are similar, you just need to mark the .tbi file as newer (replace with the correct .tbi file):
:** <code>touch ''/path/to/refVcf.vcf.gz.tbi''</code>
:* If you are using your own reference files, and you think your vcf file is newer than your .tbi file, regenerate the .tbi file using (replace with the correct .vcf.gz file):
:** <code>gotcloud/bin/tabix -f /path/to/refVcf.vcf.gz</code>

=== Targetted/Exome ===
; Exiting due to ERROR<nowiki>:</nowiki>         INVALID_SORT<nowiki>:</nowiki> ERROR<nowiki>:</nowiki> File is not coordinate sorted at record 1003         PreviousRecord was 1<nowiki>:</nowiki>15347, but this record is 1<nowiki>:</nowiki>15328

: This error occurs if you have <code>SAMTOOLS_VIEW_TARGET_ONLY = TRUE</code> in your configuration file and a read(s) in your BAM file overlaps multiple regions in your BED file.
: To fix this, set<nowiki>:</nowiki>
:: <pre>SAMTOOLS_VIEW_TARGET_ONLY = FALSE</pre>
:Further Explanation of the issue<nowiki>:</nowiki>
::The SAMTOOLS_VIEW_TARGET_ONLY set to TRUE tells our pipeline to pass to SAMTOOLS view all of the regions in the BED file that fall into a single segment of our processing.
::It appears that samtools view will then go through each of those regions independently and one at a time, selecting just the records that fall within that region.
::A problem occurs if records fall within multiple regions from the BED file. Since samtools view process the regions independently, it will write those records multiple times and they will end up out of order.

::For example<nowiki>:</nowiki>
::: If we have BED with regions chr20 20300049-20300050 and chr20 20300052-20300053
:::The resulting call to samtools view looks like this:
::: <pre>gotcloud/bin/samtools-hybrid view -q 20 -F 0x0704 bamFile.bam 20:20300049-20300050 20:20300052-20300053</pre>

:::Say there are reads at positions 20299969 and 20299994 that are each 76 bases long, they will end up being output twice by the samtools view call and in this order:
::::20299969
::::20299994
::::20299969
::::20299994

=== Does Not Run ===
;"make_path" is not exported by the <nowiki>File::Path</nowiki> module Can't continue after import errors at /home/chengu/program/gotcloud/bin/umake.pl line 21 BEGIN failed--compilation aborted at /home/chengu/program/gotcloud/bin/umake.pl line 21.

: This error occurs if you have an older version of Perl's <code>File::Path</code> module.
::Two solutions
::# Upgrade to the latest Perl File::Path module.
::# Replace all occurrances of <code>make_path</code> in <code>gotcloud/bin/</code> and <code>gotcloud/scripts/</code> with <code>mkpath</code>:
::#* '''gotcloud/bin/umake.pl''' - required to run snpcall
::#* '''gotcloud/scripts/vcfSplitChr.pl''' - required to run snpcall
::#* gotcloud/bin/genomestrip.pl - you only need to update this one if you want to run genomestrip
::#* gotcloud/scripts/bed-diff.pl - you only need to update this one if you plan to run bed-diff.pl

== ldrefine Pipeline FAQs ==
; make<nowiki>: *** [outdir/beagle/chr#/split/bgl.#.chr#.PASS.#.vcf.gz.tbi] Error 1</nowiki>
: Check the log file at: <code>outdir/beagle/chr#/split/bgl.#.err</code>
: If it says:
:: '''Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: -2'''
::: '''at phaser.j.a(Unknown Source) at phaser.j.c(Unknown Source) at phaser.h.a(Unknown Source) at phaser.E.a(Unknown Source) at phaser.Y.a(Unknown Source) at phaser.PhaseMain.<init>(Unknown Source) at phaser.PhaseMain.main(Unknown Source)'''
: You need to change the seed for running beagle.
:* In your configuration file, set:
:** <code>BEAGLE = java -Xmx4g -jar $(BIN_DIR)/beagle.20101226.jar seed=993470 gprobs=true niterations=50 lowmem=true</code>
:* If it fails again, change the seed in the above command to something else, and rerun.

== Tutorial FAQs ==

= Version Problems =

== Prior to Version 1.16 ==
; perl is not found in <code>/usr/bin/perl</code>
: Modify all of the .pl scripts in gotcloud/, gotcloud/bin/, and gotcloud/scripts/
:* If the first line contains: <code>#!/usr/bin/perl</code>
:** Replace it with: <code>#!/usr/bin/env perl</code>
:* Also, remove <code>-w</code> if it occurs on that first line of the perl scripts.
:** If it is not already specified in the script, add the following line later in the script: <code>use warnings;</code>
: This issues has been fixed in version 1.16.

== Version 1.08 ==
; ERROR<nowiki>:</nowiki> Could not find umake/'''outputDir''' in '''currentDirectory'''
: To work around this bug, specify an absolute path for OUT_DIR/--outdir
: This is due to the aligner trying to use OUT_PREFIX to transform the output directory to an absolute path, by prepending OUT_PREFIX. But, OUT_PREFIX is a snpcall setting that sets the snpcall Makefile filename. OUT_PREFIX will be deprecated in the next release. The current OUT_PREFIX which refers to the snpcall Makefile filename will be renamed to something else.

== Version 1.07 ==

;Type of arg 1 to keys must be hash (not hash element) at gotcloud/bin/Conf.pm line 155, near "}) " Compilation failed in require at gotcloud/bin/align.pl line 56.

: To fix this bug, you need to edit gotcloud/bin/Conf.pm, line 155. This error only occurs with certain versions of perl.
:Change
:<syntaxhighlight lang="perl">foreach my $key (keys $CONF_HASH{$section}) {</syntaxhighlight>
:to
:<syntaxhighlight lang="perl">foreach my $key (keys %{$CONF_HASH{$section}}) {</syntaxhighlight>

;Illegal division by zero at gotcloud/scripts/vcf-summary line 388. make<nowiki>:</nowiki> *** [vcfs/chrY/chrY.hardfiltered.vcf.gz.OK] Error 255 make<nowiki>:</nowiki> *** Waiting for unfinished jobs.... Commands finished in xxxxx secs WITH ERRORS. Check the logs
: The current version does not gracefully handle a chromosome with no variants.
: By default, GotCloud assumes all samples are female, producing no chr Y variants, which will produce this error.

: Work arounds:
::* Disable Chromosome Y by adding the following to your configuration file:
::: <pre>CHRS = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 X</pre>
::: You may also want to disable chr X since it may be slightly misleading due to the assumption that all samples are female.
::* Specify a pedigree file that tells GotCloud which individuals are Male & which are Female by adding the following to your configuration file:
::: <pre>PED_INDEX = pedigreeFile.ped</pre>
::: GotCloud only looks at the 2nd & 5th columns of the pedigree file (white-space delimited)
:::* 2nd column is the individual ID which should match the 1st column of the BAM_INDEX file.
:::* 5th column is the sex, use: 1 for male and 2 for female.

GotCloud

2015-05-26T16:52:34Z

Mktrost:

__TOC__

== Genomes on the Cloud (GotCloud) Introduction==

To handle the increasing volume of next generation sequencing and genotyping data, we created and developed software pipelines called '''Genomes on the Cloud (GotCloud).'''

GotCloud contains Mapping & Variant Calling Pipelines.

Key Features:
* Connects sequence analysis tools together in automated pipeline
** Alignment, quality control, variant calling
* Robust against unexpected system failure using GNU make
** easy restart after failure
* Massively parallel, can run hundreds of jobs
** Splits large jobs into many pieces
** Simplifies running on clusters
* Scalable to tens of thousands of samples
* Easy to use - Automates series of configurable steps
** user doesn't have to understand/configure/know the many tools required to create high quality results
* Available on Amazon Web Services (AWS) Elastic Compute Cloud (EC2)
* Run on local machines/clusters

GotCloud incorporates the alignment and variant calling pipelines that we have been using at UM into one easy to use, publicly available tool. GotCloud can run on a user's computer, on an instance in a
compute cloud, and/or can split the work up onto a cluster of machines or instances.

[[File:Gotcloud.puzzles.v2.png|500px]]

=== Getting Help with GotCloud ===

Please join in the [http://groups.google.com/group/GotCloud GotCloud Google Group] to ask / discuss / comment about these pipelines.

Currently the "join" button appears to be missing. Click "NEW TOPIC", then select "Join this group". You can then cancel the message post (or post a message).

See [[GotCloud: FAQs]] if you have any questions. If your questions are not answered there, join the google group and ask, or email Mary Kate Wing (mktrost@umich.edu).

=== Sequence Analysis Background Information ===

There are many essential steps in the analysis of next generation sequence data.

Next generation sequence data analysis starts with [http://en.wikipedia.org/wiki/FASTQ_format FASTQ files], the typical format provided from your sequencing center containing the sequence & base quality information for your data.

The fastq files are processed using the [[GotCloud: Alignment Pipeline|alignment pipeline]] which finds the most likely genomic location for each read and stores that information in a [[BAM|BAM (Binary Sequence Alignment/Map format) file]]. In addition to the sequence and base quality information contained in FASTQ files, a BAM file also contains the genomic location and some additional information about the mapping. As part of the [[GotCloud: Alignment Pipeline|alignment pipeline]], the base qualities are adjusted to more accurately reflect the likelihood that the base is correct.

The [[GotCloud: Alignment Pipeline|alignment pipeline]] can be skipped if you already have Deduped and Recalibrated BAM files. If you have BAMs, but they needed to be deduped and recalibrated, you can use our [[GotCloud:_Alignment_Sub-Pipelines#recabQC_2|recabQC pipeline]].

The [[GotCloud: Variant Calling Pipeline|variant calling pipeline]] processes the deduped and recalibrated BAM files produced by the alignment pipeline or that you provide it, generating an initial list of polymorphic sites and genotypes stored in a [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) file]. The [[GotCloud: Variant Calling Pipeline|variant calling pipeline]] then filters the variants using both hard filters and a [[SVM Filtering|Support Vector Machine (SVM)]]. It then uses haplotype information to refine these genotypes in an updated VCF file.

After completing the GotCloud Variant Calling Pipeline, [[EPACTS|EPACTS (Efficient and Parallelizable Association Container Toolbox)]] can be used to perform statistical tests to identify genome-wide association from sequence data.

[[File:GotCloudDiagram.jpg|500px]]

== Publication ==
If you use GotCloud, please cite our publication:
[http://genome.cshlp.org/content/early/2015/04/14/gr.176552.114.abstract Jun, Goo, et al. "An efficient and scalable analysis framework for variant extraction and refinement from population scale DNA sequence data." Genome research (2015): gr-176552.]

== GotCloud Setup ==

You may run the GotCloud software in several modes:
* On your own hardware running Ubuntu or Redhat/CentOS. See the instructions about installing the software below.
* On an Amazon Elastic Compute Cloud using Ubuntu or Redhat/CentOS if you do not have your own set of machines.
** See [[GotCloud: Amazon]] for more information.
** You can run on an EC2 cluster instance created by StarCluster.

GotCloud has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS. While it should work on other Linux systems, they have not yet been tested.

=== GotCloud on Amazon ===
You can take advantage of GotCloud AMI's when running on Amazon. The GotCloud already includes GotCloud and default reference files.

See [[GotCloud: Amazon]] for instructions on using GotCloud on Amazon.

=== GotCloud Setup on Any Linux Machine ===

==== GotCloud Dependencies ====

GotCloud requires certain things to be installed in order to run:
* perl - gotcloud is a perl script and it calls many other perl scripts
** Zlib.pm - required for perl scripts to read compressed files.
* make - GNU make is used to run the pipelines
* java - required to run the beagle step of the ld-aware genotype refinement
* curses/ncurses (required for samtools)
** On Ubuntu: <code>sudo apt-get install libncurses5 libncurses5-dev</code>
* cmake (required for premo)
** On Ubuntu: <code>sudo apt-get install cmake</code>

You can check if your system has the proper software installed by invoking the command
[gotcloud_path]/scripts/check_requirements.sh

==== Install GotCloud Software ====

You can install gotCloud on your system as (follow the links for the appropriate instructions):
* [[GotCloud: Source Releases|source release]] - contains the scripts and uncompiled source
* [[GotCloud: Binary Releases|binary release]] - contains the scripts and pre-compiled binaries (no source)

[[GotCloud: Versions]] describes the changes added to each version.

Alternatively if you are using Amazon EC2, you can use one of the following sets of instructions:
* Create a machine instance based on the AMI we provide: [[Amazon Single Node]]
* Create an EC2 cluster instance using [[StarCluster|StarCluster]]
For more information on using GotCloud on Amazon, see: [[GotCloud: Amazon]]

For more information on Amazon Web Services, see: https://aws.amazon.com/

==== GotCloud Reference/Resource Files ====
In order to run gotCloud, you need to provide Genetic Reference and Resource Files.

These include information about the reference sequence and dbnsp positions.

See: [[GotCloud: Genetic Reference and Resource Files]] for information about the required files. It contains a description of the required files, information about generating your own versions, as well as a downloadable set of files.
* When running on Amazon, a default set of reference files are included in the GotCloud AMI.

==== Configure GotCloud ====
* [[Configure GotCloud|Configure Gotcloud]] for your installation

== Running GotCloud Software ==

* [[GotCloud: Alignment Pipeline|Alignment Pipeline]]
** [[GotCloud: Alignment Sub-Pipelines|Alignment Sub-Pipelines]] - for if you do not want to run the entire Alignment Pipeline
* [[GotCloud: Variant Calling Pipeline|Variant Calling Pipeline]]
* [[GotCloud: Indel Calling Pipeline|Indel Calling Pipeline]]
* [[GotCloud: GenomeSTRiP Pipeline|GenomeSTRiP Pipeline]] (Structural Variation) - ''Coming Soon''
* [[GotCloud: MEI Calling Pipeline|MEI Calling Pipeline]] - ''Coming Soon''

You can also create your own pipelines. Instructions are here:
* [[GotCloud: Creating a New Pipeline]]

=== GotCloud Demos ===
GotCloud Demos (originally from our sequencing workshop):
* [[SeqShop: Sequence Mapping and Assembly Practical]]
* [[SeqShop: Variant Calling and Filtering for SNPs Practical]]
* [[SeqShop: Variant Calling and Filtering for INDELs Practical]]
* [[SeqShop: Analysis of Structural Variation Practical]]

GotCloud on Amazon Demo (snpcall & indel):
* [[GotCloud: Amazon Demo]]

Deprecated: [[Tutorial: GotCloud]]

== UMich Development/Release How-To Notes ==
* [[Releasing GotCloud]]
* Amazon EC2
** [[Creating an AMI on EC2]]
** [[Creating a Snapshot on EC2]] (deprecated)
** [[Mount S3 Volume]]
** Notes on sequence data preparation in [[Amazon Storage|Amazon Storage]].

* [[Git_FAQs#Subtrees|Upgrade Git Subtree]]