Changes

From Genome Analysis Wiki
Jump to navigationJump to search

BamUtil: dedup (view source)

Revision as of 22:31, 8 April 2019

4,093 bytes added ,  22:31, 8 April 2019
→‎Required Parameters
Line 1: Line 1: −
[[Category:C++]]
+
[[Category:BamUtil|validate]]
−
[[Category:libStatGen]]
+
[[Category:BAM Software]]
−
[[Category:libStatGen BAM]]
+
[[Category:Software]]
−
 
  −
='''COMING SOON, May/June, 2012'''=
      
= Overview of the <code>dedup</code> function of <code>[[bamUtil]]</code> =
 
= Overview of the <code>dedup</code> function of <code>[[bamUtil]]</code> =
Line 12: Line 10:  
NOTE: This tool does not properly work on templates that have more than 2 segments.  It does not properly match reads when more than 2 reads have the same read name.
 
NOTE: This tool does not properly work on templates that have more than 2 segments.  It does not properly match reads when more than 2 reads have the same read name.
    +
NOTE: Dedup cannot read from stdin since it reads the input file twice.
    
Potential future features:
 
Potential future features:
−
* Soft clip overlapping reads (for now, use: [[BamUtil: clipOverlap]]
+
* Soft clip overlapping reads (for now, use: [[BamUtil: clipOverlap]])
    
==Handling Duplicates==
 
==Handling Duplicates==
   −
The deduper reads all the alignments in a coordinate-sorted SAM/BAM looking for duplicates.
+
The deduper reads all the alignments in a coordinate-sorted SAM/BAM looking for duplicates, failing if the file is not coordinate-sorted.
   −
The deduper assumes that duplicates in the input BAM file are not marked.   
+
The deduper assumes that duplicates in the input BAM file are not marked.  When the deduper detects a marked duplicate in the input BAM file, it will throw an error and stop.  To override this behavior, use the [[#Ignore Previous Duplicate Marking (--force)|<code>--force</code>]] option;  in this mode, alignments that are marked as duplicates in the input file are unmarked before the deduper begins its detection algorithm.  The result is that only duplicates detected by the deduper will be marked in or removed from the output file.
   −
When the deduper detects a marked duplicate in the input BAM file, it will throw an error and stop.  To override this behavior, use the --force option;  in this mode, alignments that are marked as duplicates in the input file are unmarked before the deduper begins its detection algorithm.  The result is that only duplicates detected by the deduper will be marked in or removed from the output file.
+
The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate.  If a mate is not found at the expected position, an error message is printed (once per file) indicating this error.  Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads.  Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate.  If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.
   −
The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate. If a mate is not found at the expected position, an error message is printed (once per file) indicating this error.  Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads.  Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate.  If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.  Use the <code>--oneChrom</code> option to treat reads with a mate on a different chromosome as single-ended.  This option is useful if you are running the deduper on just a single chromosome.
+
With the default settings this tool should produce similar results as Picard.
 +
 
 +
Use the [[#Treat Reads with Mates On Different Chromosomes As Single-Ended (--oneChrom)|<code>--oneChrom</code>]] option to treat reads with a mate on a different chromosome as single-ended.  This option is useful if you are running the deduper on just a single chromosome. The code will use less memory with this option if mates are found on different chromosomes.  (Picard does not specially handle mates on different chromosomes, so the --oneChrom option may produce different results than Picard.)
      Line 33: Line 34:  
# Chromosome
 
# Chromosome
 
# Orientation (forward/reverse)
 
# Orientation (forward/reverse)
−
# unclipped start(forward)/end(reverse)
+
# Unclipped Start(forward)/End(reverse)
 
# Library
 
# Library
    
Rules:
 
Rules:
 
* Skip Unmapped Reads, they are not marked as duplicate
 
* Skip Unmapped Reads, they are not marked as duplicate
 +
* Reads whose mate is unmapped are treated as single-end
 
* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:
 
* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:
−
*# A paired-end record has the same key (even if the pair is not proper/the mate is unmapped/the mate is not found)<br/>-OR-
+
*# A paired-end record has the same key (even if the pair is not proper/the mate is not found)<br/>-OR-
−
*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record)
+
*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])
 
* Mark both Paired-End Reads Duplicate if:
 
* Mark both Paired-End Reads Duplicate if:
−
# Another paired-end pair has the same set of keys and has a higher base quality sum.
+
# Another paired-end pair has the same set of keys and has a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)|<code>--minBaseQual</code>]])
 
   
 
   
 
This code assumes that at most 1000 bases are clipped at the start of a read.
 
This code assumes that at most 1000 bases are clipped at the start of a read.
    +
 +
Deduping requires two passes through the file, so cannot read from stdin.
    
==Handling Recalibration==
 
==Handling Recalibration==
−
Reads skipped for recalibrating (both for determining recalibrated values and for being modified):
+
See [[BamUtil: recab]] for recalibration details.
−
* Duplicates
+
 
−
* Unmapped
+
Recalibration parameters can be applied to deduping when --recab is specified.
−
* Mapping Quality = 0
      
== How to use it ==
 
== How to use it ==
Line 60: Line 63:     
The output SAM/BAM file is also required, [[#output File (--out)|output File (--out)]].
 
The output SAM/BAM file is also required, [[#output File (--out)|output File (--out)]].
 +
 +
 +
Recommended usage with Recalibrator:
 +
 +
/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40
 +
 +
 +
Recommended usage without Recalibrator:
 +
 +
/usr/cluster/bin/bam dedup --in ${INPUT}.bam --out ${OUTPUT}.bam --force --oneChrom
    
= Usage =
 
= Usage =
−
  ./bam dedup (options) --in <InputBamFile> --out <OutputBamFile> [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--verbose] [--noeof] [--params] [--recab] --refFile <ReferenceFile> [--dbsnp <dbsnpFile>] [--blended <weight>]  
+
  ./bam dedup --in <InputBamFile> --out <OutputBamFile> [--minQual <minPhred>] [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--excludeFlags <flag>] [--verbose] [--noeof] [--params] [--recab]
 +
 
 +
Additional Recalibration Usage is documented at [[BamUtil: recab#Usage|BamUtil: recab -> Usage]]
    
= Parameters =
 
= Parameters =
 
<pre>
 
<pre>
 
Required parameters :
 
Required parameters :
−
--in <infile>  : input BAM file name (must be sorted)
+
--in <infile>  : Input BAM file name (must be sorted)
−
--out <outfile> : output BAM file name (same order with original file)
+
--out <outfile> : Output BAM file name (same order with original file)
−
Optional parameters : (see SAM format specification for details)
+
Optional parameters :  
−
--log <logfile> : log and summary statistics (default: [outfile].log)
+
--minQual <int> : Only add scores over this phred quality when determining a read's quality (default: 15)
 +
--log <logfile> : Log and summary statistics (default: [outfile].log, or stderr if --out starts with '-')
 
--oneChrom      : Treat reads with mates on different chromosomes as single-ended.
 
--oneChrom      : Treat reads with mates on different chromosomes as single-ended.
 
--rmDups        : Remove duplicates (default is to mark duplicates)
 
--rmDups        : Remove duplicates (default is to mark duplicates)
−
--force        : Allow mark-duplicated BAM file and force unmarking the duplicates
+
--force        : Allow an already mark-duplicated BAM file, unmarking any previously marked
−
                    Default is to throw errors when trying to run a mark-duplicated BAM
+
                  duplicates and apply this duplicate marking logic.  Default is to throw errors
 +
                  and exit when trying to run on an already mark-duplicated BAM
 +
--excludeFlags <flag>    : exclude reads with any of these flags set when determining or marking duplicates
 +
                          by default (0xB04): exclude unmapped, secondary reads, QC failures, and supplementary reads
 
--verbose      : Turn on verbose mode
 
--verbose      : Turn on verbose mode
−
--noeof        : do not expect an EOF block on a bam file.
+
--noeof        : Do not expect an EOF block on a bam file.
−
--params        : print the parameter settings
+
--params        : Print the parameter settings
 
--recab        : Recalibrate in addition to deduping
 
--recab        : Recalibrate in addition to deduping
 +
</pre>
 +
{{PhoneHomeParamDesc}}
   −
Recab Specific Required Parameters
+
Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters|BamUtil: recab -> Parameters]]
−
--refFile <reference file>    : reference file name
  −
Recab Specific Optional Parameters :  
  −
--dbsnp <known variance file> : dbsnp file of positions
  −
--blended <weight>            : blended model weight
  −
</pre>
     −
{{inBAMInputFile}}
+
== Required Parameters ==
 +
{{inBAMInputFile|noStdin=1}}
    
Note: The input file must be sorted by coordinate.
 
Note: The input file must be sorted by coordinate.
 
{{outBAMOutputFile}}
 
{{outBAMOutputFile}}
    +
== Optional Parameters==
 +
=== Minimum Quality for Quality Calculations (<code>--minQual</code>) ===
 +
 +
When duplicate reads are encountered, the read with the highest quality is kept.
 +
 +
To determine the quality of a read, all of the phred base quality scores above the <code>--minQual</code> value are added together.  If <code>--minQual</code> is not specified, it is defaulted to 15.
 +
 +
=== Output log & Summary Statistics FileName (<code>--log</code>) ===
 +
 +
Output file name for writing logs & summary statistics.
 +
 +
If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log".  Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr.  If the filename after --log starts with '-' it will write to stderr.
 +
 +
=== Treat Reads with Mates On Different Chromosomes As Single-Ended (<code>--oneChrom</code>) ===
 +
 +
If a read's mate is not found it will not be used for duplicate marking.  If you are running on a single chromosome, all read's whose mates are on different chromosomes will not be used for duplicate marking.  The <code>--oneChrom</code> option will treat reads with mates on a different chromosome as single-ended.
 +
 +
=== Remove Duplicates (<code>--rmDups</code>) ===
 +
 +
Instead of marking a read as duplicate in the flag, the <code>--rmDups</code> option will remove it from the output BAM file. 
 +
 +
=== Ignore Previous Duplicate Marking (<code>--force</code>) ===
 +
 +
By default the deduper will throw an error and stop if a read is already marked as duplicate.  The <code>--force</code> option will removes any previous duplicate marking and marks the reads from scratch.  The resulting output file will only have reads determined by the deduper marked as duplicates.
 +
 +
=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
 +
Skip records with any of the specified flags set, default 0xB04
 +
 +
By default skips reads with any of the following flags set:
 +
* unmapped
 +
* secondary alignment
 +
* fails QC checks
 +
* supplementary reads
 +
 +
Secondary (0x100) and Supplementary (0x800) reads currently must be excluded.
 +
 +
=== Turn on Verbose Mode (<code>--verbose</code>) ===
 +
 +
Turn on verbose logging to get more log messages in the log and to stderr.
    
{{noeofBGZFParameter}}
 
{{noeofBGZFParameter}}
 
{{paramsParameter}}
 
{{paramsParameter}}
 +
 +
== Recalibrate (<code>--recab</code>) ==
 +
 +
This option will recalibrate the input file in addition to deduping.
 +
 +
See [[BamUtil: recab]] for recalibration details.
 +
 +
{{PhoneHomeParameters}}
    
= Return Value =
 
= Return Value =
Line 100: Line 164:  
Returns -1 if input parameters are invalid.
 
Returns -1 if input parameters are invalid.
   −
Returns the SamStatus for the reads/writes (0 on success).
+
Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).
Retrieved from "http://genome.sph.umich.edu/wiki/Special:MobileDiff/4903...15092"

Navigation menu