BamUtil: dedup - Revision history

Mktrost: /* Required Parameters */

2019-04-09T02:31:36Z

Required Parameters

← Older revision		Revision as of 02:31, 9 April 2019
Line 104:		Line 104:

	== Required Parameters ==		== Required Parameters ==
	{{inBAMInputFile}}		{{inBAMInputFile\|noStdin=1}}

	Note: The input file must be sorted by coordinate.		Note: The input file must be sorted by coordinate.

Mktrost: /* Overview of the dedup function of bamUtil */

2019-04-09T02:03:08Z

Overview of the dedup function of bamUtil

← Older revision		Revision as of 02:03, 9 April 2019
Line 10:		Line 10:
	NOTE: This tool does not properly work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.		NOTE: This tool does not properly work on templates that have more than 2 segments. It does not properly match reads when more than 2 reads have the same read name.

			NOTE: Dedup cannot read from stdin since it reads the input file twice.

	Potential future features:		Potential future features:

Mktrost: /* Implementation Notes */

2019-04-09T02:02:12Z

Implementation Notes

← Older revision		Revision as of 02:02, 9 April 2019
Line 46:		Line 46:

	This code assumes that at most 1000 bases are clipped at the start of a read.		This code assumes that at most 1000 bases are clipped at the start of a read.


			Deduping requires two passes through the file, so cannot read from stdin.

	==Handling Recalibration==		==Handling Recalibration==

Mktrost: /* Implementation Notes */

2016-04-15T02:30:35Z

Implementation Notes

← Older revision		Revision as of 02:30, 15 April 2016
Line 38:		Line 38:
	Rules:		Rules:
	* Skip Unmapped Reads, they are not marked as duplicate		* Skip Unmapped Reads, they are not marked as duplicate
			* Reads whose mate is unmapped are treated as single-end
	* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:		* Mark a Single-End Read Duplicate (or remove it if configured to do so) if:
	*# A paired-end record has the same key (even if the pair is not proper~~/the mate is unmapped~~/the mate is not found)<br/>-OR-		*# A paired-end record has the same key (even if the pair is not proper/the mate is not found)<br/>-OR-
	*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)\|<code>--minBaseQual</code>]])		*# A single-end record has the same key and a higher base quality sum (sum of all base qualities in the record above [[#Minimum Quality for Quality Calculations (--minQual)\|<code>--minBaseQual</code>]])
	* Mark both Paired-End Reads Duplicate if:		* Mark both Paired-End Reads Duplicate if:

Mktrost: /* Skip Records with any of the Specified Flags (--excludeFlags) */

2016-03-06T04:45:48Z

Skip Records with any of the Specified Flags (--excludeFlags)

← Older revision		Revision as of 04:45, 6 March 2016
Line 138:		Line 138:
	* supplementary reads		* supplementary reads

	~~The Deduper will not work if secondary or supplementary~~ reads ~~are not~~ excluded~~. It will not properly find the mates since there will be more than 2 reads with the same mate information~~.		Secondary (0x100) and Supplementary (0x800) reads currently must be excluded.

	=== Turn on Verbose Mode (<code>--verbose</code>) ===		=== Turn on Verbose Mode (<code>--verbose</code>) ===

Mktrost: /* Skip Records with any of the Specified Flags (--excludeFlags) */

2016-03-06T04:20:58Z

Skip Records with any of the Specified Flags (--excludeFlags)

← Older revision		Revision as of 04:20, 6 March 2016
Line 138:		Line 138:
	* supplementary reads		* supplementary reads

	~~This parameter was added in version 1~~.~~0.10~~.		The Deduper will not work if secondary or supplementary reads are not excluded. It will not properly find the mates since there will be more than 2 reads with the same mate information.

	=== Turn on Verbose Mode (<code>--verbose</code>) ===		=== Turn on Verbose Mode (<code>--verbose</code>) ===

Mktrost: /* Parameters */

2016-03-06T04:15:36Z

Parameters

← Older revision		Revision as of 04:15, 6 March 2016
Line 88:		Line 88:
	and exit when trying to run on an already mark-duplicated BAM		and exit when trying to run on an already mark-duplicated BAM
	--excludeFlags <flag> : exclude reads with any of these flags set when determining or marking duplicates		--excludeFlags <flag> : exclude reads with any of these flags set when determining or marking duplicates
	by default (~~0x304~~): exclude unmapped, secondary reads, ~~and~~ QC failures		by default (0xB04): exclude unmapped, secondary reads, QC failures, and supplementary reads
	--verbose : Turn on verbose mode		--verbose : Turn on verbose mode
	--noeof : Do not expect an EOF block on a bam file.		--noeof : Do not expect an EOF block on a bam file.
Line 130:		Line 130:

	=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===		=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
	Skip records with any of the specified flags set, default ~~0x304~~		Skip records with any of the specified flags set, default 0xB04

	By default skips reads with any of the following flags set:		By default skips reads with any of the following flags set:
Line 136:		Line 136:
	* secondary alignment		* secondary alignment
	* fails QC checks		* fails QC checks
			* supplementary reads

	This parameter was added in version 1.0.10.		This parameter was added in version 1.0.10.

Mktrost at 18:05, 6 January 2014

2014-01-06T18:05:49Z

← Older revision		Revision as of 18:05, 6 January 2014
Line 94:		Line 94:
	--recab : Recalibrate in addition to deduping		--recab : Recalibrate in addition to deduping
	</pre>		</pre>
			{{PhoneHomeParamDesc}}

	Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters\|BamUtil: recab -> Parameters]]		Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters\|BamUtil: recab -> Parameters]]

Mktrost at 17:29, 6 January 2014

2014-01-06T17:29:49Z

← Older revision		Revision as of 17:29, 6 January 2014
Line 70:		Line 70:

	= Usage =		= Usage =
	./bam dedup --in <InputBamFile> --out <OutputBamFile> [--minQual <minPhred>] [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--verbose] [--noeof] [--params] [--recab]		./bam dedup --in <InputBamFile> --out <OutputBamFile> [--minQual <minPhred>] [--log <logFile>] [--oneChrom] [--rmDups] [--force] [--excludeFlags <flag>] [--verbose] [--noeof] [--params] [--recab]

	Additional Recalibration Usage is documented at [[BamUtil: recab#Usage\|BamUtil: recab -> Usage]]		Additional Recalibration Usage is documented at [[BamUtil: recab#Usage\|BamUtil: recab -> Usage]]
Line 87:		Line 87:
	duplicates and apply this duplicate marking logic. Default is to throw errors		duplicates and apply this duplicate marking logic. Default is to throw errors
	and exit when trying to run on an already mark-duplicated BAM		and exit when trying to run on an already mark-duplicated BAM
			--excludeFlags <flag> : exclude reads with any of these flags set when determining or marking duplicates
			by default (0x304): exclude unmapped, secondary reads, and QC failures
	--verbose : Turn on verbose mode		--verbose : Turn on verbose mode
	--noeof : Do not expect an EOF block on a bam file.		--noeof : Do not expect an EOF block on a bam file.
Line 95:		Line 97:
	Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters\|BamUtil: recab -> Parameters]]		Additional Recalibration Parameters are documented at [[BamUtil: recab#Parameters\|BamUtil: recab -> Parameters]]

			== Required Parameters ==
	{{inBAMInputFile}}		{{inBAMInputFile}}

Line 100:		Line 103:
	{{outBAMOutputFile}}		{{outBAMOutputFile}}

	== Minimum Quality for Quality Calculations (<code>--minQual</code>) ==		== Optional Parameters==
			=== Minimum Quality for Quality Calculations (<code>--minQual</code>) ===

	When duplicate reads are encountered, the read with the highest quality is kept.		When duplicate reads are encountered, the read with the highest quality is kept.
Line 106:		Line 110:
	To determine the quality of a read, all of the phred base quality scores above the <code>--minQual</code> value are added together. If <code>--minQual</code> is not specified, it is defaulted to 15.		To determine the quality of a read, all of the phred base quality scores above the <code>--minQual</code> value are added together. If <code>--minQual</code> is not specified, it is defaulted to 15.

	== Output log & Summary Statistics FileName (<code>--log</code>) ==		=== Output log & Summary Statistics FileName (<code>--log</code>) ===

	Output file name for writing logs & summary statistics.		Output file name for writing logs & summary statistics.
Line 112:		Line 116:
	If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.		If this parameter is not specified, it will write to the output file specified in <code>--out</code> + ".log". Or if the output bam is written to stdout (<code>--out</code> starts with '-'), the logs will be written to stderr. If the filename after --log starts with '-' it will write to stderr.

	== Treat Reads with Mates On Different Chromosomes As Single-Ended (<code>--oneChrom</code>) ==		=== Treat Reads with Mates On Different Chromosomes As Single-Ended (<code>--oneChrom</code>) ===

	If a read's mate is not found it will not be used for duplicate marking. If you are running on a single chromosome, all read's whose mates are on different chromosomes will not be used for duplicate marking. The <code>--oneChrom</code> option will treat reads with mates on a different chromosome as single-ended.		If a read's mate is not found it will not be used for duplicate marking. If you are running on a single chromosome, all read's whose mates are on different chromosomes will not be used for duplicate marking. The <code>--oneChrom</code> option will treat reads with mates on a different chromosome as single-ended.

	== Remove Duplicates (<code>--rmDups</code>) ==		=== Remove Duplicates (<code>--rmDups</code>) ===

	Instead of marking a read as duplicate in the flag, the <code>--rmDups</code> option will remove it from the output BAM file.		Instead of marking a read as duplicate in the flag, the <code>--rmDups</code> option will remove it from the output BAM file.

	== Ignore Previous Duplicate Marking (<code>--force</code>) ==		=== Ignore Previous Duplicate Marking (<code>--force</code>) ===

	By default the deduper will throw an error and stop if a read is already marked as duplicate. The <code>--force</code> option will removes any previous duplicate marking and marks the reads from scratch. The resulting output file will only have reads determined by the deduper marked as duplicates.		By default the deduper will throw an error and stop if a read is already marked as duplicate. The <code>--force</code> option will removes any previous duplicate marking and marks the reads from scratch. The resulting output file will only have reads determined by the deduper marked as duplicates.

	== Turn on Verbose Mode (<code>--verbose</code>) ==		=== Skip Records with any of the Specified Flags (<code>--excludeFlags</code>)===
			Skip records with any of the specified flags set, default 0x304

			By default skips reads with any of the following flags set:
			* unmapped
			* secondary alignment
			* fails QC checks

			This parameter was added in version 1.0.10.

			=== Turn on Verbose Mode (<code>--verbose</code>) ===

	Turn on verbose logging to get more log messages in the log and to stderr.		Turn on verbose logging to get more log messages in the log and to stderr.
Line 136:		Line 150:

	See [[BamUtil: recab]] for recalibration details.		See [[BamUtil: recab]] for recalibration details.

			{{PhoneHomeParameters}}

	= Return Value =		= Return Value =
Line 141:		Line 157:
	Returns -1 if input parameters are invalid.		Returns -1 if input parameters are invalid.

	Returns the SamStatus for the reads/writes (0 on success).		Returns the SamStatus for the reads/writes (0 on success, non-0 on failure).

Mktrost at 22:43, 8 January 2013

2013-01-08T22:43:50Z

← Older revision		Revision as of 22:43, 8 January 2013
Line 22:		Line 22:
	The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate. If a mate is not found at the expected position, an error message is printed (once per file) indicating this error. Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads. Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate. If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.		The handling of paired-end reads assumes that the mate information in the SAM/BAM records is accurate. If a mate is not found at the expected position, an error message is printed (once per file) indicating this error. Paired-end reads whose mate cannot be found are not marked duplicate and are not used for duplicate marking of other paired-end reads. Single-end reads with the same key as paired-end reads whose mate cannot be found are still marked as duplicate. If this error is encountered, you may want to fix the mate information and reprocess the file through the deduper.

	Use the [[#Treat Reads with Mates On Different Chromosomes As Single-Ended (--oneChrom)\|<code>--oneChrom</code>]] option to treat reads with a mate on a different chromosome as single-ended. This option is useful if you are running the deduper on just a single chromosome. The code will use less memory with this option if mates are found on different chromosomes.		With the default settings this tool should produce similar results as Picard.

			Use the [[#Treat Reads with Mates On Different Chromosomes As Single-Ended (--oneChrom)\|<code>--oneChrom</code>]] option to treat reads with a mate on a different chromosome as single-ended. This option is useful if you are running the deduper on just a single chromosome. The code will use less memory with this option if mates are found on different chromosomes. (Picard does not specially handle mates on different chromosomes, so the --oneChrom option may produce different results than Picard.)


Line 60:		Line 62:
	Recommended usage with Recalibrator:		Recommended usage with Recalibrator:

	/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} ~~--skipFit~~ --oneChrom --storeQualTag OQ --maxBaseQual 40		/usr/cluster/bin/bam dedup --recab --in ${INPUT}.bam --out ${OUTPUT}.bam --force --refFile ${REF} --dbsnp ${DBSNP} --oneChrom --storeQualTag OQ --maxBaseQual 40