Genome Analysis Wiki - User contributions [en]

TOPMed Site Visit 2018

2018-09-28T14:14:36Z

Tblackw: Replaced content with " '''Successfully occurred on Thursday September 13, 2018'''"

'''Successfully occurred on Thursday September 13, 2018'''

TOPMed Site Visit 2018

2018-09-04T15:43:55Z

Tblackw:

'''One day: Thursday, September 13, 2018, 8:30 am - 4:00 pm ?'''

Perhaps 5 1/2 total hours of presentations.

= Draft Agenda =

== Introduction and Overview of TOPMed data resources and services (30 minutes) ==

''Goncalo Abecasis will present overview of IRC activies''

* Introduce IRC personnel and their expertise
* Sequence for 130,000+ participants
* Variant calls and genotypes, phased and unphased
* Structural variant calls in progress
* BRAVO variant browser
* ENCORE analysis server
* TOPMed imputation reference panel
* Main developments in the past year, including security improvements, Manual, etc.

== Characteristics of variants in data freeze 6 (20 minutes) ==

''Hyun Min Kang and Jonathan LeFaive will present this section''

* Overall numbers
* Differences by ancestry, study and sequencing center
* Allele frequencies of deleterious variants
* Genotype accuracy
* Compare harmonized versus sequencing center mappings
* Process for variant calling, genotyping, filtering, phasing and distribution
* Process for interim 'snapshot' genotypes

== Calling structural variants (20 minutes) ==

''Our colleagues at Baylor will present this section. William Salerno?''

* Procedures and plans for structural variant calling
* Benefits of ensemble approach
* Initial results
* Data access mechanisms
* Anticipated data size
* Coordination with SNPs and indels

== BRAVO variant browser helps to assess variant quality (20 minutes) ==

''Daniel Taliun will present this section''

* Purpose
* Which studies are included
* Usage / main features
* Both rare and common variants
* Improvements in the last year
* Access via an applications programming interface (API)
* View all information used in filtering
* Coordination with gnomAD
* Potential plans for PheWeb integration

== Break (30 minutes) ==

== Summary of contract spending to date (20 minutes) ==

''Denise Bianchi and Goncalo Abecasis will present this section''

* Broad subdivisions: personnel, cloud storage, cloud computing, hardware
* Divided between Task 1 and Task 2

== Improved results from the latest TOPMed imputation panel (20 minutes) ==

''Goncalo Abecasis will present this, unless Lukas Forer is available. Ketian Yu to prepare summaries of imputation quality''

[[Media:nhlbi.4761.imputation.accuracy.2018aug31.pptx|'''(slides)''']]

* Principle of operation
* Measuring imputation quality in different populations
* Pushing the low frequency boundary
* Improved accuracy for African American and Latino samples
* Challenges and opportunities from collaboration and integration with NIH Commons / NHLBI Stage

== Potential Population Genetics Update (20 minutes) ==

'' Check with Sebastian Zoellner''

== Cloud access to TOPMed sequence data (20 minutes) ==

''Tom Blackwell to take the lead on this section''

[[Media:nhlbi.4768.fusera.slides.01.pdf|'''(slides)''']]

* NCBI's 'Fusera' controlled access mechanism
* User perspective -- involves a Google or Amazon billing project
* What is needed for users to have a great overall experience?
* Education and training for users

== Lunch (70 minutes) ==

== ENCORE analysis server (20 minutes) ==

''Matthew Flickinger will present this section''

* Principle of operation
* Releases only aggregate data summaries
* Visualizations help to assess results
* Data sharing and collaboration tools
* Capability to re-run previous jobs with new data
* SAIGE analysis gives accurate results in case-control studies
* Usage statistics comparing last 12-months to previous 12-months
* Some highlights from user survey results

== Manuscript support (30 minutes) ==

''Albert Vernon Smith will present overview for this section''

* How can and how is the IRC supporting TOPMed manuscripts and discoveries?

* Overall TOPMed landmark paper
* Analysis of telomere length
* Mitochondrial DNA copy number
* Lipids analysis using TOPMed imputed genotypes
* UK BioBank with TOPMed imputation
* Context specific mutation rates
* Data sharing with Centers for Common Disease Genetics

== Interactions with outside groups (30 minutes) ==

''Albert Vernon Smith to take the lead on this section''

* NIH Data Commons
* NHLBI Data STAGE
* NHGRI Centers for Common Disease Genetics
* Global Alliance for Genomics and Health (GA4GH)
* NIMH Parkinsons Disease Consortium

== Break (20 minutes) ==

== Future plans for the next Task Order (30 minutes) ==

== Feedback from NHLBI (40 minutes) ==

== Finish (3:50 pm) ==

TOPMed Site Visit 2018

2018-08-29T18:18:18Z

Tblackw: Created page with "'''One day: Thursday, September 13, 2018, 8:30 am - 3:00 pm ?''' Perhaps 4 1/3 total hours of presentations, if lunch is brought in. (If you think I'm a bit optimistic o..."

'''One day: Thursday, September 13, 2018, 8:30 am - 3:00 pm ?'''

Perhaps 4 1/3 total hours of presentations, if lunch is brought in.

(If you think I'm a bit optimistic on the timings, I'd have to agree with you.)

= Draft Agenda =

== Introduction and Overview of TOPMed data resources and services (30 minutes) ==

''Goncalo Abecasis will present overview of IRC activies''

* Introduce IRC personnel and their expertise
* Sequence for 130,000+ participants
* Variant calls and genotypes, phased and unphased
* Structural variant calls in progress
* BRAVO variant browser
* ENCORE analysis server
* TOPMed imputation reference panel
* Main developments in the past year, including security improvements, Manual, etc.

== Characteristics of variants in data freeze 6 (20 minutes) ==

''Hyun Min Kang and Jonathan LeFaive will present this section''

* Overall numbers
* Differences by ancestry, study and sequencing center
* Allele frequencies of deleterious variants
* Genotype accuracy
* Compare harmonized versus sequencing center mappings
* Process for variant calling, genotyping, filtering, phasing and distribution
* Process for interim 'snapshot' genotypes

== Calling structural variants (20 minutes) ==

''Our colleagues at Baylor will present this section. William Salerno?''

* Procedures and plans for structural variant calling
* Benefits of ensemble approach
* Initial results
* Data access mechanisms
* Anticipated data size
* Coordination with SNPs and indels

== BRAVO variant browser helps to assess variant quality (20 minutes) ==

''Daniel Taliun will present this section''

* Purpose
* Which studies are included
* Usage / main features
* Both rare and common variants
* Improvements in the last year
* Access via an applications programming interface (API)
* View all information used in filtering
* Coordination with gnomAD
* Potential plans for PheWeb integration

== Break (30 minutes) ==

== ENCORE analysis server (15 minutes) ==

''Matthew Flickinger will present this section''

* Principle of operation
* Releases only aggregate data summaries
* Visualizations help to assess results
* Data sharing and collaboration tools
* Capability to re-run previous jobs with new data
* SAIGE analysis gives accurate results in case-control studies
* Usage statistics comparing last 12-months to previous 12-months
* Some highlights from user survey results

== Improved results from the latest TOPMed imputation panel (15 minutes) ==

''Goncalo Abecasis will present this, unless Lukas Forrer is available. Ketian to prepare summaries of imputation quality''

* Principle of operation
* Measuring imputation quality in different populations
* Pushing the low frequency boundary
* Improved accuracy for African American and Latino samples
* Challenges and opportunities from collaboration and integration with NIH Commons / NHLBI Stage

== Potential Population Genetics Update (15 minutes) ==

'' Check with Sebastian Zoellner''

== Manuscript support (30 minutes) ==

''Albert Vernon Smith will present overview for this section''

* How can and how is the IRC supporting TOPMed manuscripts and discoveries?

* Overall TOPMed landmark paper
* Analysis of telomere length
* Mitochondrial DNA copy number
* Lipids analysis using TOPMed imputed genotypes
* UK BioBank with TOPMed imputation
* Context specific mutation rates
* Data sharing with Centers for Common Disease Genetics

== Lunch (45 minutes) ==

== Interactions with outside groups (30 minutes) ==

''Albert Vernon Smith to take the lead on this section''

* NIH Data Commons
* NHLBI Data STAGE
* NHGRI Centers for Common Disease Genetics
* Global Alliance for Genomics and Health (GA4GH)
* NIMH Parkinsons Disease Consortium

== Cloud access to TOPMed sequence data (15 minutes) ==

''Tom Blackwell to take the lead on this section''

* NCBI's 'Fusera' controlled access mechanism
* User perspective -- involves a Google or Amazon billing project
* What is needed for users to have a great overall experience?
* Education and training for users

== Feedback from NHLBI (60 minutes) ==

Generic Exome Analysis Plan

2010-04-21T12:01:52Z

Tblackw:

This page outlines a generic plan for analysis of a whole exome sequencing project. The idea is that the points listed here might serve as a starting point for discussion of the analyses needed in a specific project.

The initial version of this document was prepared with input from Shamil Sunayev, Ron Do and Goncalo Abecasis.

= Read Mapping and Variant Calling =

The first step in any analysis is to map sequence reads, callibrate base qualities, and call variants. Even at this stage, some simple quality metrics can be evaluated and will help identify potentially problematic samples.

== Prior to Mapping ==

; Evaluate Base Composition Along Reads
: Calculate the proportion of A, C, G, T bases along each read. Flag runs with evidence of unusual patterns of base composition compared to the target genome.

; Evaluate Machine Quality Scores Along Reads
: Calculate average quality scores per position. Flag runs with evidence of unusual quality score distributions.

; Calculate Number of Reads
: Calculate the input number of reads and number of bases for each sequenced sample

== Read Mapping ==

; Map Reads with Appropriate Read Mapper
: Currently, [bio-bwa.sourceforge.net/bwa.shtm BWA] is a convenient, widely used read mapper.

; Mark Duplicate Reads
: Duplicate reads, whether generated as PCR artifacts during library preparation or optical duplicates during image analysis and base-calling, can mislead variant calling algorithms. To avoid problems, one typically removes from consideration all reads that appear to map at exactly the same location (for paired ends, only reads for which both ends map to the same locations are excluded.)

; Basic Mapping Statistics
: We should tally the overall proportion of mapped reads.
: We should also tally the proportion of reads that map
:* Inside the target regions
:* Near the target regions (defined as within 200bp of each target)
:* Elsewhere in the genome (defined as regions that are >200bp from each target)

; Recalibrate Base Quality Scores
: Base quality scores can be updated by comparing sites that are unlikely to vary (such as those not currently reported as variants in dbSNP or in the most recent [[1000 Genome Project]] analyses.

; Update Base Quality Score Metrics
: Generate new curves with base quality scores per position.
: Calculate the number of mapped bases that reach at least Q20. Potentially, calculate Q20 ''equivalent'' bases by summing the quality scores for bases with base quality >Q20 and dividing the total by 20.

; Evaluate Coverage as Function of GC Content
: For each target region, calculate read depth and also the proportion of GC bases in the reference genome. Flag samples where coverage varies strongly as a function of GC content.

== Verify Sample Identities ==

; Verify that Each Sequenced Sample Matches Prior Information
: We should verify that each sequenced sample matches previous genotypes for that sample. Ideally, this should be done using a likelihood based approach that can also identify potentially contaminated samples.

; Adjudicate Mislabeled Samples
: If any samples that don't match prior genotype data are encountered, the read data for these samples should be compared to all available samples to identify potential sample mixups.

; Identify Potentially Related Samples
: Perhaps this step should be done *after* variant calling?

== Generate Variant Calls ==

; Generate Initial Set of Variant Calls
: Variant calls should be generated taking into account all available samples simultaneously. We should consider variants calls that fall in target regions but also those that fall near the target. An open question is whether off target variant calls will be trustworthy.

; Generate Linkage Disequilibrium Aware Set of Variant Calls
: A typical exome call set might include many sites where no variant is called due to low coverage. If we can integrate samples with previous [[GWAS]] data, we should be able to generate an update and much improved set of variant calls for each individual. Due to limitations in current calling methods, the quality of variant call sets is expected to increase substantially if multiple variant callers are used and their results are merged democratically.

; Annotate Functional Impact of Each Variant
: Called variants should be annotated according to their potential function. At a minimum, we should distinguish synonymous, non-synonymous, conserved splice site, 5'UTR, 3'UTR and other variants. Ideally, we should also assess [[SIFT]] and [[PolyPhen]] scores for conserved variants.

; Calculate Overall Frequency Spectrum
: Calculate observed frequency spectrum and compare to neutral expectations (which are that the number of variants should be roughly proportional to ''1/n'', where ''n'' is the number of minor alleles).

; Annotate Overall Variant Characteristics
: Calculate overall ratio of transitions to transversions, separately for coding and non-coding variants. Within coding variants, analyse synonymous and non-synonymous variants separately.

; CpG Sites
: Calculate the rate of per base pair heterozygosity at potential CpG sites and compare this to other sites.

; Tabulate for Each Sample
:* The number of synonymous and non-synonymous variants
:* The number of unique and shared variants
:* The number of transitions and transversions

== Evaluate Variant Calls for Reference Samples ==

; If Previously Sequenced Samples Available, Assess Variant Calls There
: Assess concordance rate at non-reference sites

; If Duplicate Samples are Available, Assess Concordance
: Assess concordance rate at non-reference sites.

; If Nuclear Families are Available, Assess Mendelian Consistency
: When reporting rates of Mendelian inconsistencies, report these not as a fraction of all sites, but as a fraction of sites with at least one non-reference call in the trio.

== Variant Filters ==

Initial sets of SNP calls invariably will include many false positives. The fraction of false positives among all variants called is likely to increase as more and more samples are sequenced. To keep the fraction of false positives under control, it is important to both apply an increasingly strict set of quality control filters but also to experimentally validate some newly discovered variants. Many of these filters are currently implemented in [[GATK]].

; Mapping Quality Filter
: Consider removing variants at sites with low mapping quality scores. Even if the average mapping quality score for a site is high, consider removing variants at sites where a noticeable fraction of reads have low mapping quality scores.

; Allele Balance Filter
: Among individuals who are assigned an heterozygous genotype, check the proportion of reads supporting each allele. Consider filtering out variants where one of the alleles accounts for <30% of reads.

; Local Realignment Filter
: Consider removing single nucleotide variants at sites where local realignment of all covering sequence reads suggests that a length polymorphism is present in the population.

; Read Depth
: Consider filtering out variants at sites where total read depth is unusually low. Unfortunately, capture protocols introduce very large amounts of variation in sequencing depth and it is usually not possible to accurately filter out sites sequenced at very high depth.

= Special Considerations for Admixed Samples =

; Estimate Local Ancestry Using GWAS Data
: For studies that include admixed samples, we should estimate local ancestry using GWAS data. If GWAS are not available, it is strongly recommended that these data should be generated. In principle, local ancestry estimates can be generated even before exome sequencing is complete.

; Estimate Global Ancestry Covariates Using PCA or MDS Analysis

= Initial Association Analyses =

We anticipate that, at least early on, the initial association analysis of whole exome datasets in the context of complex trait association studies will focus on identifying and resolving quality control issues that might result in unexpected artifacts.

== Initial Single SNP Tests ==

In principle, these tests only have power for common variants. In practice, particularly when permutation based methods are used to assess significance, there should be little loss in power by testing all variants using single SNP tests. These tests should include:

; Logistic Regression Based Tests for Discrete Traits
: Discrete outcomes should be evaluated using logistic regression. It is important to include appropriate covariates. For most traits, these might include age and sex and, potentially, principal components of ancestry.

; Linear Regression Based Tests for Quantitative Traits
: For quantitative traits that are not strongly selected, linear regression based tests can also be used. Again, it is important to include an appropriate set of covariates. For many quantitative traits, it may be a very good idea to normalize traits to minimize the impact of outliers on association results.

; Using genotypes as outcomes for Selected Quantitative Traits
: For both discrete and quantitative traits, analysis can be repeated using genotypes (scored as 0, 1 and 2) as outcomes and phenotypes as predictors.

== Q-Q Plots ==

After carrying out initial single SNP tests, generate Q-Q plots for each analysis. Verify that Q-Q plots are reasonable and that genomic control value is close to 1.0. If not, refine sample and variant filters as needed.

== Burden Tests ==

The same analyses that were originally carried for single variants should be carried out for groups of rare variants. In principle, one could simply use the presence of a rare variant (or a particular class of rare variant, such as a non-synonymous variant or a newly discovered variant) as a predictor and repeat the logistic regression, linear regression or genotype regression described above. For an initial pass, I think the precise form of this analysis is not critical, because the next step is to...

== More Q-Q Plots ==

After carrying out initial burden tests, generate Q-Q plots for each analysis. Verify that Q-Q plots are reasonable and that genomic control value is close to 1.0. If not, refine sample and variant filters as needed.

= Visualize Results =

A number of displays will likely be useful. Probably these should include:

* Manhattan Plots
* [[LocusZoom]] Plots
* Q-Q Plots

= Think You Are Done? =

'''No way!!!'''

== Indels and Structural Variants ==

You still need a plan to call and evaluate short insertions and deletions as well as larger structural variants.

== Pathway Based Analyses ==

Carry out analyses that include groups of genes with similar biological function (for example, according to [[Gene Ontology]] or [[Kyoto Encyclopedia of Genes and Genomes]] annotations.

Generic Exome Analysis Plan

2010-04-21T11:54:41Z

Tblackw:

This page outlines a generic plan for analysis of a whole exome sequencing project. The idea is that the points listed here might serve as a starting point for discussion of the analyses needed in a specific project.

The initial version of this document was prepared with input from Shamil Sunayev, Ron Do and Goncalo Abecasis.

= Read Mapping and Variant Calling =

The first step in any analysis is to map sequence reads, callibrate base qualities, and call variants. Even at this stage, some simple quality metrics can be evaluated and will help identify potentially problematic samples.

== Prior to Mapping ==

; Evaluate Base Composition Along Reads
: Calculate the proportion of A, C, G, T bases along each read. Flag runs with evidence of unusual patterns of base composition compared to the target genome.

; Evaluate Machine Quality Scores Along Reads
: Calculate average quality scores per position. Flag runs with evidence of unusual quality score distributions.

; Calculate Number of Reads
: Calculate the input number of reads and number of bases for each sequenced sample

== Read Mapping ==

; Map Reads with Appropriate Read Mapper
: Currently, [bio-bwa.sourceforge.net/bwa.shtm BWA] is a convenient, widely used read mapper.

; Mark Duplicate Reads
: Duplicate reads, whether generated as PCR artifacts during library preparation or optical duplicates during image analysis and base-calling, can mislead variant calling algorithms. To avoid problems, one typically removes from consideration all reads that appear to map at exactly the same location (for paired ends, only reads for which both ends map to the same locations are excluded.)

; Basic Mapping Statistics
: We should tally the overall proportion of mapped reads.
: We should also tally the proportion of reads that map
:* Inside the target regions
:* Near the target regions (defined as within 200bp of each target)
:* Elsewhere in the genome (defined as regions that are >200bp from each target)

; Recalibrate Base Quality Scores
: Base quality scores can be updated by comparing sites that are unlikely to vary (such as those not currently reported as variants in dbSNP or in the most recent [[1000 Genome Project]] analyses.

; Update Base Quality Score Metrics
: Generate new curves with base quality scores per position.
: Calculate the number of mapped bases that reach at least Q20. Potentially, calculate Q20 ''equivalent'' bases by summing the quality scores for bases with base quality >Q20 and dividing the total by 20.

; Evaluate Coverage as Function of GC Content
: For each target region, calculate read depth and also the proportion of GC bases in the reference genome. Flag samples where coverage varies strongly as a function of GC content.

== Verify Sample Identities ==

; Verify that Each Sequenced Sample Matches Prior Information
: We should verify that each sequenced sample matches previous genotypes for that sample. Ideally, this should be done using a likelihood based approach that can also identify potentially contaminated samples.

; Adjudicate Mislabeled Samples
: If any samples that don't match prior genotype data are encountered, the read data for these samples should be compared to all available samples to identify potential sample mixups.

; Identify Potentially Related Samples
: Perhaps this step should be done *after* variant calling?

== Generate Variant Calls ==

; Generate Initial Set of Variant Calls
: Variant calls should be generated taking into account all available samples simultaneously. We should consider variants calls that fall in target regions but also those that fall near the target. An open question is whether off target variant calls will be trustworthy.

; Generate Linkage Disequilibrium Aware Set of Variant Calls
: A typical exome call set might include many sites where no variant is called due to low coverage. If we can integrate samples with previous [[GWAS]] data, we should be able to generate an update and much improved set of variant calls for each individual. Due to limitations in current calling methods, the quality of variant call sets is expected to increase substantially if multiple variant callers are used and their results are merged democratically.

; Annotate Functional Impact of Each Variant
: Called variants should be annotated according to their potential function. At a minimum, we should distinguish synonymous, non-synonymous, conserved splice site, 5'UTR, 3'UTR and other variants. Ideally, we should also assess [[SIFT]] and [[PolyPhen]] scores for conserved variants.

; Calculate Overall Frequency Spectrum
: Calculate observed frequency spectrum and compare to neutral expectations (which are that the number of variants should be roughly proportional to ''1/n'', where ''n'' is the number of minor alleles).

; Annotate Overall Variant Characteristics
: Calculate overall ratio of transitions to transversions, separately for coding and non-coding variants. Within coding variants, analyse synonymous and non-synonymous variants separately.

; CpG Sites
: Calculate the rate of per base pair heterozygosity at potential CpG sites and compare this to other sites.

; Tabulate for Each Sample
:* The number of synonymous and non-synonymous variants
:* The number of unique and shared variants
:* The number of transitions and transversions

== Evaluate Variant Calls for Reference Samples ==

; If Previously Sequenced Samples Available, Assess Variant Calls There
: Assess concordance rate at non-reference sites

; If Duplicate Samples are Available, Assess Concordance
: Assess concordance rate at non-reference sites.

; If Nuclear Families are Available, Assess Mendelian Consistency
: When reporting rates of Mendelian inconsistencies, report these not as a fraction of all sites, but as a fraction of sites with at least one non-reference call in the trio.

== Variant Filters ==

Initial sets of SNP calls invariably will include many false positives. The fraction of false positives among all variants called is likely to increase as more and more samples are sequenced. To keep the fraction of false positives under control, it is important to both apply an increasingly strict set of quality control filters but also to experimentally validate some newly discovered variants. Many of these filters are currently implemented in [[GATK]].

; Mapping Quality Filter
: Consider removing variants at sites with low mapping quality scores. Even if the average mapping quality score for a site is high, consider removing variants at sites where a noticeable fraction of reads have low mapping quality scores.

; Allele Balance Filter
: Among individuals who are assigned an heterozygous genotype, check the proportion of reads supporting each allele. Consider filtering out variants where one of the alleles accounts for <30% of reads.

; Local Realignment Filter
: Consider removing single nucleotide variants at sites where local realignment of all covering sequence reads suggests that a length polymorphism is present in the population.

; Read Depth
: Consider filtering out variants at sites where total read depth is unusually low. Unfortunately, capture protocols introduce very large amounts of variation in sequencing depth and it is usually not possible to accurately filter out sites sequenced at very high depth.

= Special Considerations for Admixed Samples =

; Estimate Local Ancestry Using GWAS Data
: For studies that include admixed samples, we should estimate local ancestry using GWAS data. If GWAS are not available, it is strongly recommended that these data should be generated. In principle, local ancestry estimates can be generated even before exome sequencing is complete.

; Estimate Global Ancestry Covariates Using PCA or MDS Analysis

= Initial Association Analyses =

We anticipate that, at least early on, the initial association analysis of whole exome datasets in the context of complex trait association studies will focus on identifying and resolving quality control issues that might result in unexpected artifacts.

== Initial Single SNP Tests ==

In principle, these tests only have power for common variants. In practice, particularly when permutation based methods are used to assess significance, there should be little loss in power by testing all variants using single SNP tests. These tests should include:

; Logistic Regression Based Tests for Discrete Traits
: Discrete outcomes should be evaluated using logistic regression. It is important to include appropriate covariates. For most traits, these might include age and sex and, potentially, principal components of ancestry.

; Linear Regression Based Tests for Quantitative Traits
: For quantitative traits that are not strongly selected, linear regression based tests can also be used. Again, it is important to include an appropriate set of covariates. For many quantitative traits, it may be a very good idea to normalize traits to minimize the impact of outliers on association results.

; Using genotypes as outcomes for Selected Quantitative Traits
: For both discrete and quantitative traits, analysis can be repeated using genotypes (scored as 0, 1 and 2) as outcomes and phenotypes as predictors.

== Q-Q Plots ==

After carrying out initial single SNP tests, generate Q-Q plots for each analysis. Verify that Q-Q plots are reasonable and that genomic control value is close to 1.0. If not, refine sample and variant filters as needed.

== Burden Tests ==

The same analyses that were originally carried for single variants should be carried out for groups of rare variants. In principle, one could simple use the presence of a rare variant (or a particular class of rare variant, such as a non-synonymous variant or a newly discovered variant) as a predictor and repeat the logistic regression, linear regression or genotype regression described above. For an initial pass, I think the precise form of this analysis is not critical, because the next step is to...

== More Q-Q Plots ==

After carrying out initial burden tests, generate Q-Q plots for each analysis. Verify that Q-Q plots are reasonable and that genomic control value is close to 1.0. If not, refine sample and variant filters as needed.

= Visualize Results =

A number of displays will likely be useful. Probably these should include:

* Manhattan Plots
* [[LocusZoom]] Plots
* Q-Q Plots

= Think You Are Done? =

'''No way!!!'''

== Indels and Structural Variants ==

You still need a plan to call and evaluate short insertions and deletions as well as larger structural variants.

== Pathway Based Analyses ==

Carry out analyses that include groups of genes with similar biological function (for example, according to [[Gene Ontology]] or [[Kyoto Encyclopedia of Genes and Genomes]] annotations.

Generic Exome Analysis Plan

2010-04-21T11:51:20Z

Tblackw:

This page outlines a generic plan for analysis of a whole exome sequencing project. The idea is that the points listed here might serve as a starting point for discussion of the analyses needed in a specific project.

The initial version of this document was prepared with input from Shamil Sunayev, Ron Do and Goncalo Abecasis.

= Read Mapping and Variant Calling =

The first step in any analysis is to map sequence reads, callibrate base qualities, and call variants. Even at this stage, some simple quality metrics can be evaluated and will help identify potentially problematic samples.

== Prior to Mapping ==

; Evaluate Base Composition Along Reads
: Calculate the proportion of A, C, G, T bases along each read. Flag runs with evidence of unusual patterns of base composition compared to the target genome.

; Evaluate Machine Quality Scores Along Reads
: Calculate average quality scores per position. Flag runs with evidence of unusual quality score distributions.

; Calculate Number of Reads
: Calculate the input number of reads and number of bases for each sequenced sample

== Read Mapping ==

; Map Reads with Appropriate Read Mapper
: Currently, [bio-bwa.sourceforge.net/bwa.shtm BWA] is a convenient, widely used read mapper.

; Mark Duplicate Reads
: Duplicate reads, whether generated as PCR artifacts during library preparation or optical duplicates during image analysis and base-calling, can mislead variant calling algorithms. To avoid problems, one typically removes from consideration all reads that appear to map at exactly the same location (for paired ends, only reads for which both ends map to the same locations are excluded.)

; Basic Mapping Statistics
: We should tally the overall proportion of mapped reads.
: We should also tally the proportion of reads that map
:* Inside the target regions
:* Near the target regions (defined as within 200bp of each target)
:* Elsewhere in the genome (defined as regions that are >200bp from each target)

; Recalibrate Base Quality Scores
: Base quality scores can be updated by comparing sites that are unlikely to vary (such as those not currently reported as variants in dbSNP or in the most recent [[1000 Genome Project]] analyses.

; Update Base Quality Score Metrics
: Generate new curves with base quality scores per position.
: Calculate the number of mapped bases that reach at least Q20. Potentially, calculate Q20 ''equivalent'' bases by summing the quality scores for bases with base quality >Q20 and dividing the total by 20.

; Evaluate Coverage as Function of GC Content
: For each target region, calculate read depth and also the proportion of GC bases in the reference genome. Flag samples where coverage varies strongly as a function of GC content.

== Verify Sample Identities ==

; Verify that Each Sequenced Sample Matches Prior Information
: We should verify that each sequenced sample matches previous genotypes for that sample. Ideally, this should be done using a likelihood based approach that can also identify potentially contaminated samples.

; Adjudicate Mislabeled Samples
: If any samples that don't match prior genotype data are encountered, the read data for these samples should be compared to all available samples to identify potential sample mixups.

; Identify Potentially Related Samples
: Perhaps this step should be done *after* variant calling?

== Generate Variant Calls ==

; Generate Initial Set of Variant Calls
: Variant calls should be generated taking into account all available samples simultaneously. We should consider variants calls that fall in target regions but also those that fall near the target. An open question is whether off target variant calls will be trustworthy.

; Generate Linkage Disequilibrium Aware Set of Variant Calls
: A typical exome call set might include many sites where no variant is called due to low coverage. If we can integrate samples with previous [[GWAS]] data, we should be able to generate an update and much improved set of variant calls for each individual. Due to limitations in current calling methods, the quality of variant call sets is expected to increase substantially if multiple variant callers are used and their results are merged democratically.

; Annotate Functional Impact of Each Variant
: Called variants should be annotated according to their potential function. At a minimum, we should distinguish synonymous, non-synonymous, conserved splice site, 5'UTR, 3'UTR and other variants. Ideally, we should also assess [[SIFT]] and [[PolyPhen]] scores for conserved variants.

; Calculate Overall Frequency Spectrum
: Calculate observed frequency spectrum and compare to neutral expectations (which are that the number of variants should be roughly proportional to ''1/n'', where ''n'' is the number of minor alleles).

; Annotate Overall Variant Characteristics
: Calculate overall ratio of transitions to transversions, separately for coding and non-coding variants. Within coding variants, analyse synonymous and non-synonymous variants separately.

; CpG Sites
: Calculate the rate of per base pair heterozygosity at potential CpG sites and compare this to other sites.

; Tabulate for Each Sample
:* The number of synonymous and non-synonymous variants
:* The number of unique and shared variants
:* The number of transitions and transversions

== Evaluate Variant Calls for Reference Samples ==

; If Previously Sequenced Samples Available, Assess Variant Calls There
: Assess concordance rate at non-reference sites

; If Duplicate Samples are Available, Assess Concordance
: Assess concordance rate at non-reference sites.

; If Nuclear Families are Available, Assess Mendelian Consistency
: When reporting rates of Mendelian inconsistencies, report these not as a fraction of all sites, but as a fraction of sites with at least one non-reference call in the trio.

== Variant Filters ==

Initial sets of SNP calls invariably will include many false positives. The fraction of false positives among all variants called is likely to increase as more and more samples are sequenced. To keep the fraction of false positives under control, it is important to both apply an increasingly strict set of quality control filters but also to experimentally validate some newly discovered variants. Many of these filters are currently implemented in [[GATK]].

; Mapping Quality Filter
: Consider removing variants at sites with low mapping quality scores. Even if the average mapping quality score for a site is high, consider removing variants at sites where a noticeable fraction of reads have low mapping quality scores.

; Allele Balance Filter
: Among individuals who are assigned an heterozygous genotype, check the proportion of reads supporting each allele. Consider filtering out variants where one of the alleles accounts for <30% of reads.

; Local Realignment Filter
: Consider removing single nucleotide variants at sites where local realignment of all covering sequence reads suggests that a short indel polymorphism is present in the population.

; Read Depth
: Consider filtering out variants at sites where total read depth is unusually low. Unfortunately, capture protocols introduce very large amounts of variation in sequencing depth and it is usually not possible to accurately filter out sites sequenced at very high depth.

= Special Considerations for Admixed Samples =

; Estimate Local Ancestry Using GWAS Data
: For studies that include admixed samples, we should estimate local ancestry using GWAS data. If GWAS are not available, it is strongly recommended that these data should be generated. In principle, local ancestry estimates can be generated even before exome sequencing is complete.

; Estimate Global Ancestry Covariates Using PCA or MDS Analysis

= Initial Association Analyses =

We anticipate that, at least early on, the initial association analysis of whole exome datasets in the context of complex trait association studies will focus on identifying and resolving quality control issues that might result in unexpected artifacts.

== Initial Single SNP Tests ==

In principle, these tests only have power for common variants. In practice, particularly when permutation based methods are used to assess significance, there should be little loss in power by testing all variants using single SNP tests. These tests should include:

; Logistic Regression Based Tests for Discrete Traits
: Discrete outcomes should be evaluated using logistic regression. It is important to include appropriate covariates. For most traits, these might include age and sex and, potentially, principal components of ancestry.

; Linear Regression Based Tests for Quantitative Traits
: For quantitative traits that are not strongly selected, linear regression based tests can also be used. Again, it is important to include an appropriate set of covariates. For many quantitative traits, it may be a very good idea to normalize traits to minimize the impact of outliers on association results.

; Using genotypes as outcomes for Selected Quantitative Traits
: For both discrete and quantitative traits, analysis can be repeated using genotypes (scored as 0, 1 and 2) as outcomes and phenotypes as predictors.

== Q-Q Plots ==

After carrying out initial single SNP tests, generate Q-Q plots for each analysis. Verify that Q-Q plots are reasonable and that genomic control value is close to 1.0. If not, refine sample and variant filters as needed.

== Burden Tests ==

The same analyses that were originally carried for single variants should be carried out for groups of rare variants. In principle, one could simple use the presence of a rare variant (or a particular class of rare variant, such as a non-synonymous variant or a newly discovered variant) as a predictor and repeat the logistic regression, linear regression or genotype regression described above. For an initial pass, I think the precise form of this analysis is not critical, because the next step is to...

== More Q-Q Plots ==

After carrying out initial burden tests, generate Q-Q plots for each analysis. Verify that Q-Q plots are reasonable and that genomic control value is close to 1.0. If not, refine sample and variant filters as needed.

= Visualize Results =

A number of displays will likely be useful. Probably these should include:

* Manhattan Plots
* [[LocusZoom]] Plots
* Q-Q Plots

= Think You Are Done? =

'''No way!!!'''

== Indels and Structural Variants ==

You still need a plan to call and evaluate short insertions and deletions as well as larger structural variants.

== Pathway Based Analyses ==

Carry out analyses that include groups of genes with similar biological function (for example, according to [[Gene Ontology]] or [[Kyoto Encyclopedia of Genes and Genomes]] annotations.

Examples of Read Mapping with Karma and BWA

2009-12-11T15:06:48Z

Tblackw:

# Some instructions for read mapping and variant calling using the
# University of Michigan tools and procedures. Please view this as source.

# -- Paul Anderson and Tom Blackwell, December 11, 2009 --

# These instructions will cover three components:

# (1) read mapping using karma
# (2) read mapping using bwa
# (3) samtools processing from .sam to .glf

# These instructions are specifically with reference to the CEU trio
# chromosome 20 test data set that has been distributed. Please
# see some general discussion at the beginning of item 2. Everything
# that's not commented out should be runnable code, although you
# will need to alter the directory and file names.

# (1) Command line procedure for karma read mapping of the CEU trio
# chromosome 20 test data set. Paul Anderson writes:

# Start with four subdirectories:

# indiv containing single and paired end Illumina .fastq sequence files,
# ab containing AB SOLiD color space reads as in the test data set,
# k.ref containing the gzipped human genome reference sequence, and
# k.out an empty directory for the resulting karma .sam and .stats files.

# First, build karma's binary word index files:

cd k.ref
zcat human_g1k_v37.fasta.gz > human_g1k_v37.fa
karma --reference human_g1k_v37.fa --createIndex --occurrenceCutoff 5000

# then map Illumina paired end or single end reads as, for example:

cd ../k.out
karma --reference ../k.ref/human_g1k_v37.fa --pairedReads --maxInsert 2000 \
../indiv/SRR010941_1.recal.fastq.gz ../indiv/SRR010941_2.recal.fastq.gz

karma --reference ../k.ref/human_g1k_v37.fa --maxInsert 2000 \
../indiv/SRR010936.recal.fastq.gz

# The read lengths in the Illumina data I saw are long enough so that the
# only parameter worth tweaking might be --occurrenceCutoff in the index
# structures. I will need to study the mapping performance to see if this
# makes any appreciable difference in speed.

# LS 454 fastq files can also be mapped using the same index, but the
# mapping success rate will be much lower.

# For AB Solid data, a color space reference and set of index files must be
# created. For the lengths of reads I see, we will create two -- one with a
# 12-mer index, and the other with a 15-mer.

cd ../k.ref
ln -s human_g1k_v37.fa human_g1k_v37_12CS.fa
ln -s human_g1k_v37.fa human_g1k_v37_15CS.fa
karma --reference human_g1k_v37_12CS.fa --colorSpace --createIndex --wordSize 12
karma --reference human_g1k_v37_15CS.fa --colorSpace --createIndex --wordSize 15

# Then, short color space reads (25/26-mers) should be mapped using the 12-mer index:

cd ../k.out
karma --reference ../k.ref/human_g1k_v37.fa \
--csreference ../k.ref/human_g1k_v37_12CS.fa \
--colorSpace --pairedReads --maxInsert 2000 \
../ab/TG150_1.color.space.fastq.gz \
../ab/TG150_2.color.space.fastq.gz

# To map longer color space reads (>30-mer), use the bigger index:

karma --reference ../k.ref/human_g1k_v37.fa \
--csreference ../k.ref/human_g1k_v37_15CS.fa \
--colorSpace --pairedReads --maxInsert 2000 \
../ab/TG152_1.color.space.fastq.gz \
../ab/TG152_2.color.space.fastq.gz

# You can use the shorter reference to map any length reads, but it is slower.
# The 15-mer color space index (human_g1k_v37_15CS.fa) is a good compromise
# for mapping reads 30 or more bases, and for machines with 24GB of RAM.

# The maxInsert of 2000 is somewhat arbitrary - you want a number that includes
# the bulk of the paired end read insert distances you are interested in. More
# is slower, so you don't want an arbitrarily large value.

# (2) Command line procedure for bwa read mapping of the CEU trio
# chromosome 20 test data set.

# As a general principle, one wants to keep all sequence data separated
# by sequencing technology and by individual throughout the entire mapping
# and variant calling process. Different technologies may have different
# requirements for read mapping and different characteristics for variant
# calling. This can be done either using the directory structure or with a
# file tracking database.

# As a further complication, the Broad Institute Illumina sequencing runs
# in the current test data set benefit from removing leading and trailing Ns
# before read mapping with bwa. All Illumina data for the CEU trio that
# have SRR identifiers are Broad Institute runs. (There are no Broad
# runs for the YRI trio. There, SRR identifiers will indicate either Beijing
# or Wash U runs.) I have used exactly the same bwa command lines with
# or without N-trimming.

# In what follows, I will use somewhat simplified directory and file names.
# At this stage, it's easiest to have the command line prompt remain at top
# level in the directory structure, and refer to all files using relative
# pathnames, relative to the location of the prompt. At the start, suppose
# that there are two subdirectories: 'bwa.ref' containing the gzipped human
# genome reference sequence, and 'indiv' containing many .fastq files for
# the target individual generated by the appropriate sequencing technology.
# The bwa program has an inconvenient habit of writing to std.err. I routinely
# redirect that to a log file so that I can keep on working while bwa runs in
# the background.

# In general, bwa read mapping requires initial indexing of the genome
# reference sequence, followed by two passes for each .fastq file. The
# software components of this process are:

# bwa index -- Do this only when a new version of the genome reference
# sequence is released. Takes just under two hours.
# bwa aln -- Run this on every .fastq file individually. Highly
# variable timings -- takes between 10 minutes and many
# hours per .fastq file. Better data runs quicker.
# bwa samse -- Converts a single .fastq / .sai pair to .sam alignment
# format. Usually under 1 minute per file.
# bwa sampe -- Converts paired end .fastq / .sai pairs (four files total)
# to a single .sam alignment file. 1 - 2 minutes per run.
# (In general, the CEU trio chromosome 20 test data set
# contains VERY small .fastq files. Chromosome 20 is just
# over 2% of the entire genome.)

# Sample command lines. This is csh syntax.

# The parameter string " -n 0.002 -M 7 -R 25 " used in bwa aln below
# is just my initial guess. I have made an entire run using this string.
# I might make another run using Heng Li's default values for all three
# parameters and compare the results, but I have no answers from this yet.

bwa index -a bwtsw -p bwa.ref/ncbi.v37.ref \
bwa.ref/human_g1k_v37.fasta.gz >>& logfile

mkdir bwa.sai bwa.sam

( nice +20 bwa aln -n 0.002 -M 7 -R 25 bwa.ref/ncbi.v37.ref \
indiv/SRRx_1.fastq.gz > bwa.sai/SRRx_1.sai ) >>& logfile

# ... and the same for SRRx_2.fastq.gz or SRRx.fastq.gz.
# Then, depending on whether these are paired end reads or not, either:

( nice +20 bwa sampe bwa.ref/ncbi.v37.ref \
bwa.sai/SRRx_1.sai bwa.sai/SRRx_2.sai \
indiv/SRRx_1.fastq.gz indiv/SRRx_2.fastq.gz \
> bwa.sam/SRRx.pair.sam ) >>& logfile

# or, for single end reads:

( nice +20 bwa samse bwa.ref/ncbi.v37.ref \
bwa.sai/SRRx.sai indiv/SRRx.fastq.gz \
> bwa.sam/SRRx.single.sam ) >>& logfile

# For LS 454 sequence data, bwa read mapping is a one-step process.
# This uses the same genome reference sequence index as for Illumina
# data above, and does not use paired end information in the mapping.
# (The directory bwa.sam shown here should be separate from that
# created above for Illumina data. Similarly for AB SOLiD below.)

( nice +20 bwa dbwtsw bwa.ref/ncbi.v37.ref indiv/SRRx.fastq.gz \
> bwa.sam/SRRx.sam ) >>& logfile

# For AB SOLiD data, one must build a separate index structure for
# the genome reference sequence, and (completely undocumented) one
# must rewrite all of the .fastq files replacing 0,1,2,3,"." with
# A,C,G,T,N in every line of color space sequence and omitting the
# first two characters from each line of converted sequence and from
# the base call quality strings. (An awk script does this conversion
# really quickly.)

bwa index -a bwtsw -p bwa.ref/ncbi.color.ref -c \
bwa.ref/human_g1k_v37.fasta.gz >>& logfile

mkdir bwa.sai bwa.sam

( nice +20 bwa aln -n 0.002 -M 7 -R 25 -o 0 -c bwa.ref/ncbi.color.ref \
indiv/TG145_1.cvt.gz > bwa.sai/TG145_1.sai ) >>& logfile

# ... and the same for TG145_2.cvt.gz or TG145.cvt.gz.
# Then, depending on whether these are paired end reads or not, either:

( nice +20 bwa sampe bwa.ref/ncbi.color.ref \
bwa.sai/TG145_1.sai bwa.sai/TG145_2.sai \
indiv/TG145_1.cvt.gz indiv/TG145_2.cvt.gz \
> bwa.sam/SRRx.pair.sam ) >>& logfile

# or:

( nice +20 bwa samse bwa.ref/ncbi.color.ref \
bwa.sai/TG145.sai indiv/TG145.cvt.gz \
> bwa.sam/SRRx.single.sam ) >>& logfile

# (3) Here is the process which builds the .glf format files used for
# SNP calling out of .sam format files for individual sequencing runs
# produced using either read mapping algorithm. This will use only
# samtools utilities and contains nothing specific to either read mapper.

# For illustration, I will assume two of the directories left over from
# bwa read mapping in the preceding section. These are:

# bwa.ref containing files human_g1k_v37.fasta.gz, human_g1k_v37.fasta.fai, and
# bwa.sam containing all of the single-end and paired-end .sam files generated
# for one individual and sequencing technology.

# The full set of command lines is: (Note csh syntax again -- and this time,
# I will change directories for convenience in file naming.)

mkdir bwa.bam indiv.bam
cd bwa.bam
set minq=17

foreach file ( ../bwa.sam/*.sam )
set unsorted=`basename $file .sam`.nosort.bam
samtools view -bhuS -o $unsorted -q $minq $file >>& logfile
samtools sort $unsorted `basename $file .sam` >>& logfile
rm $unsorted
end

unset minq

# This is the point where one would insert steps to recalibrate
# base call quality values, check genotype identity or remove
# duplicate sequence reads. Each of these steps is specific to
# an individual sequencing run.

cd ../indiv.bam
samtools merge person.bam ../bwa.bam/*.bam
samtools index person.bam
samtools view person.bam 20 | samtools pileup \
-f ../bwa.ref/human_g1k_v37.fasta.gz \
-t ../bwa.ref/human_g1k_v37.fasta.fai \
-g - > person.glf

Examples of Read Mapping with Karma and BWA

2009-12-11T14:55:06Z

Tblackw: Created page with ' # Some instructions for read mapping and variant calling using the # University of Michigan tools and procedures. # -- Paul Anderson and Tom Blackwell, December 11, 2009…'

# Some instructions for read mapping and variant calling using the
# University of Michigan tools and procedures.

# -- Paul Anderson and Tom Blackwell, December 11, 2009 --

# These instructions will cover three components:

# (1) read mapping using karma
# (2) read mapping using bwa
# (3) samtools processing from .sam to .glf

# These instructions are specifically with reference to the CEU trio
# chromosome 20 test data set that has been distributed. Please
# see some general discussion at the beginning of item 2. Everything
# that's not commented out should be runnable code, although you
# will need to alter the directory and file names.

# (1) Command line procedure for karma read mapping of the CEU trio
# chromosome 20 test data set. Paul Anderson writes:

# Start with four subdirectories:

# indiv containing single and paired end Illumina .fastq sequence files,
# ab containing AB SOLiD color space reads as in the test data set,
# k.ref containing the gzipped human genome reference sequence, and
# k.out an empty directory for the resulting karma .sam and .stats files.

# First, build karma's binary word index files:

cd k.ref
zcat human_g1k_v37.fasta.gz > human_g1k_v37.fa
karma --reference human_g1k_v37.fa --createIndex --occurrenceCutoff 5000

# then map Illumina paired end or single end reads as, for example:

cd ../k.out
karma --reference ../k.ref/human_g1k_v37.fa --pairedReads --maxInsert 2000 \
../indiv/SRR010941_1.recal.fastq.gz ../indiv/SRR010941_2.recal.fastq.gz

karma --reference ../k.ref/human_g1k_v37.fa --maxInsert 2000 \
../indiv/SRR010936.recal.fastq.gz

# The read lengths in the Illumina data I saw are long enough so that the
# only parameter worth tweaking might be --occurrenceCutoff in the index
# structures. I will need to study the mapping performance to see if this
# makes any appreciable difference in speed.

# LS 454 fastq files can also be mapped using the same index, but the
# mapping success rate will be much lower.

# For AB Solid data, a color space reference and set of index files must be
# created. For the lengths of reads I see, we will create two -- one with a
# 12-mer index, and the other with a 15-mer.

cd ../k.ref
ln -s human_g1k_v37.fa human_g1k_v37_12CS.fa
ln -s human_g1k_v37.fa human_g1k_v37_15CS.fa
karma --reference human_g1k_v37_12CS.fa --colorSpace --createIndex --wordSize 12
karma --reference human_g1k_v37_15CS.fa --colorSpace --createIndex --wordSize 15

# Then, short color space reads (25/26-mers) should be mapped using the 12-mer index:

cd ../k.out
karma --reference ../k.ref/human_g1k_v37.fa \
--csreference ../k.ref/human_g1k_v37_12CS.fa \
--colorSpace --pairedReads --maxInsert 2000 \
../ab/TG150_1.color.space.fastq.gz \
../ab/TG150_2.color.space.fastq.gz

# To map longer color space reads (>30-mer), use the bigger index:

karma --reference ../k.ref/human_g1k_v37.fa \
--csreference ../k.ref/human_g1k_v37_15CS.fa \
--colorSpace --pairedReads --maxInsert 2000 \
../ab/TG152_1.color.space.fastq.gz \
../ab/TG152_2.color.space.fastq.gz

# You can use the shorter reference to map any length reads, but it is slower.
# The 15-mer color space index (human_g1k_v37_15CS.fa) is a good compromise
# for mapping reads 30 or more bases, and for machines with 24GB of RAM.

# The maxInsert of 2000 is somewhat arbitrary - you want a number that includes
# the bulk of the paired end read insert distances you are interested in. More
# is slower, so you don't want an arbitrarily large value.

# (2) Command line procedure for bwa read mapping of the CEU trio
# chromosome 20 test data set.

# As a general principle, one wants to keep all sequence data separated
# by sequencing technology and by individual throughout the entire mapping
# and variant calling process. Different technologies may have different
# requirements for read mapping and different characteristics for variant
# calling. This can be done either using the directory structure or with a
# file tracking database.

# As a further complication, the Broad Institute Illumina sequencing runs
# in the current test data set benefit from removing leading and trailing Ns
# before read mapping with bwa. All Illumina data for the CEU trio that
# have SRR identifiers are Broad Institute runs. (There are no Broad
# runs for the YRI trio. There, SRR identifiers will indicate either Beijing
# or Wash U runs.) I have used exactly the same bwa command lines with
# or without N-trimming.

# In what follows, I will use somewhat simplified directory and file names.
# At this stage, it's easiest to have the command line prompt remain at top
# level in the directory structure, and refer to all files using relative
# pathnames, relative to the location of the prompt. At the start, suppose
# that there are two subdirectories: 'bwa.ref' containing the gzipped human
# genome reference sequence, and 'indiv' containing many .fastq files for
# the target individual generated by the appropriate sequencing technology.
# The bwa program has an inconvenient habit of writing to std.err. I routinely
# redirect that to a log file so that I can keep on working while bwa runs in
# the background.

# In general, bwa read mapping requires initial indexing of the genome
# reference sequence, followed by two passes for each .fastq file. The
# software components of this process are:

# bwa index # Do this only when a new version of the genome reference
# sequence is released. Takes just under two hours.
# bwa aln # Run this on every .fastq file individually. Highly
# variable timings -- takes between 10 minutes and many
# hours per .fastq file. Better data runs quicker.
# bwa samse # Converts a single .fastq / .sai pair to .sam alignment
# format. Usually under 1 minute per file.
# bwa sampe # Converts paired end .fastq / .sai pairs (four files total)
# to a single .sam alignment file. 1 - 2 minutes per run.
# (In general, the CEU trio chromosome 20 test data set
# contains VERY small .fastq files. Chromosome 20 is just
# over 2% of the entire genome.)

# Sample command lines. This is csh syntax.

# The parameter string " -n 0.002 -M 7 -R 25 " used in bwa aln below
# is just my initial guess. I have made an entire run using this string.
# I might make another run using Heng Li's default values for all three
# parameters and compare the results, but I have no answers from this yet.

bwa index -a bwtsw -p bwa.ref/ncbi.v37.ref \
bwa.ref/human_g1k_v37.fasta.gz >>& logfile

mkdir bwa.sai bwa.sam

( nice +20 bwa aln -n 0.002 -M 7 -R 25 bwa.ref/ncbi.v37.ref \
indiv/SRRx_1.fastq.gz > bwa.sai/SRRx_1.sai ) >>& logfile

# ... and the same for SRRx_2.fastq.gz or SRRx.fastq.gz.
# Then, depending on whether these are paired end reads or not, either:

( nice +20 bwa sampe bwa.ref/ncbi.v37.ref \
bwa.sai/SRRx_1.sai bwa.sai/SRRx_2.sai \
indiv/SRRx_1.fastq.gz indiv/SRRx_2.fastq.gz \
> bwa.sam/SRRx.pair.sam ) >>& logfile

# or, for single end reads:

( nice +20 bwa samse bwa.ref/ncbi.v37.ref \
bwa.sai/SRRx.sai indiv/SRRx.fastq.gz \
> bwa.sam/SRRx.single.sam ) >>& logfile

# For LS 454 sequence data, bwa read mapping is a one-step process.
# This uses the same genome reference sequence index as for Illumina
# data above, and does not use paired end information in the mapping.
# (The directory bwa.sam shown here should be separate from that
# created above for Illumina data. Similarly for AB SOLiD below.)

( nice +20 bwa dbwtsw bwa.ref/ncbi.v37.ref indiv/SRRx.fastq.gz \
> bwa.sam/SRRx.sam ) >>& logfile

# For AB SOLiD data, one must build a separate index structure for
# the genome reference sequence, and (completely undocumented) one
# must rewrite all of the .fastq files replacing 0,1,2,3,"." with
# A,C,G,T,N in every line of color space sequence and omitting the
# first two characters from each line of converted sequence and from
# the base call quality strings. (An awk script does this conversion
# really quickly.)

bwa index -a bwtsw -p bwa.ref/ncbi.color.ref -c \
bwa.ref/human_g1k_v37.fasta.gz >>& logfile

mkdir bwa.sai bwa.sam

( nice +20 bwa aln -n 0.002 -M 7 -R 25 -o 0 -c bwa.ref/ncbi.color.ref \
indiv/TG145_1.cvt.gz > bwa.sai/TG145_1.sai ) >>& logfile

# ... and the same for TG145_2.cvt.gz or TG145.cvt.gz.
# Then, depending on whether these are paired end reads or not, either:

( nice +20 bwa sampe bwa.ref/ncbi.color.ref \
bwa.sai/TG145_1.sai bwa.sai/TG145_2.sai \
indiv/TG145_1.cvt.gz indiv/TG145_2.cvt.gz \
> bwa.sam/SRRx.pair.sam ) >>& logfile

# or:

( nice +20 bwa samse bwa.ref/ncbi.color.ref \
bwa.sai/TG145.sai indiv/TG145.cvt.gz \
> bwa.sam/SRRx.single.sam ) >>& logfile

# (3) Here is the process which builds the .glf format files used for
# SNP calling out of .sam format files for individual sequencing runs
# produced using either read mapping algorithm. This will use only
# samtools utilities and contains nothing specific to either read mapper.

# For illustration, I will assume two of the directories left over from
# bwa read mapping in the preceding section. These are:

# bwa.ref containing files human_g1k_v37.fasta.gz, human_g1k_v37.fasta.fai, and
# bwa.sam containing all of the single-end and paired-end .sam files generated
# for one individual and sequencing technology.

# The full set of command lines is: (Note csh syntax again -- and this time,
# I will change directories for convenience in file naming.)

mkdir bwa.bam indiv.bam
cd bwa.bam
set minq=17

foreach file ( ../bwa.sam/*.sam )
set unsorted=`basename $file .sam`.nosort.bam
samtools view -bhuS -o $unsorted -q $minq $file >>& logfile
samtools sort $unsorted `basename $file .sam` >>& logfile
rm $unsorted
end

unset minq

# This is the point where one would insert steps to recalibrate
# base call quality values, check genotype identity or remove
# duplicate sequence reads. Each of these steps is specific to
# an individual sequencing run.

cd ../indiv.bam
samtools merge person.bam ../bwa.bam/*.bam
samtools index person.bam
samtools view person.bam 20 | samtools pileup \
-f ../bwa.ref/human_g1k_v37.fasta.gz \
-t ../bwa.ref/human_g1k_v37.fasta.fai \
-g - > person.glf

Main Page

2009-12-11T14:54:37Z

Tblackw: /* Sequence Analysis Tools */

{| style="width:100%; background:#fcfcfc; margin-top:1.2em; border:1px solid #ccc;" |
| style="width:100%; text-align:center; white-space:nowrap; color:#000;" |
<div style="font-size:162%; border:none; margin:0; padding:.1em; color:#000;">Abecasis Group Wiki</div>
|}

[[Image:2009.08_Group_Retreat_Photo.jpg|400px|center|Group Photo]]

== Welcome! ==

Welcome to our brand new wiki!

If you would like to contribute, [[Special:UserLogin|log-in]] or [http://csgwiki.sph.umich.edu/index.php?title=Special:UserLogin&type=signup create an account]. We recommend using your e-mail address or Michigan uniqname as your user id.

For basic instructions, see [http://en.wikipedia.org/wiki/Wikipedia:Tutorial the Wikipedia Tutorial].

== Sequence Analysis Tools ==

We are developing several tools for the analysis of next generation sequence data.

=== Read Mapping ===

[[Karma|Karma]] - Our fast short read aligner

[[Karma-colorspace|Karma-ColorSpace]] - QUICKSTART on mapping color space reads

[[Examples|Examples]] - Sample command lines with discussion

=== Variant Calling ===

[[glfSingle]] - Variant calling for a single, deeply sequenced individual

[[glfTrio]] - Variant calling for a single, deeply sequenced nuclear family with two parents and one child

[[glfMultiples]] -- Variant calling for multiple, unrelated individuals

=== Variant Annotation ===

[[vcfCodingSnps]] -- Annotate coding variants in a VCF file.

Karma-colorspace

2009-12-02T20:19:38Z

Tblackw:

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

* A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})

* A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])

* Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)

* Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])

* Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]])

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference). Suppose that NCBI36.fa is a FASTA file which contains the nucleotide sequences for all chromosomes.

The command to invoke is:

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a packed binary sequence file and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  With a .csfastq   file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam format named   "single.sam"  derived from the .fastq  file name. 
  

Multiple input files are also acceptable and will produce multiple .sam output files, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.1.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files, etc. and write output files  "pair.1.sam", "pair.3.sam", etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

Keep in mind that KARMA requires color space reads that are at least twice as long as the index word size plus two (including the leading primer base).  (For nucleotide space, the minimum read length is twice the word size.)  For example, KARMA uses an index word size of 15 by default, so it will only map color space reads that are 32 colors or longer (including the primer base). 

== Auxiliary tools ==

The ABI SOLiD platform generates separate FASTA and quality files named  XXX.csfasta  and  XXX\_QV.qual.  We provide a script  ''solid2csfastq.py''  which converts these into a single color space FASTQ file named  XXX.csfastq.  We believe that a single color space FASTQ file simplifies post processing. 

== Choose an appropriate size for word index ==

The length of the index words influences mapping performance.  Using short index words increases the number of calculation cycles for a single read and duplications of a single word.  On the other side, long index words require much larger memory.  Please also keep in mind that appropriate size is related to your hardware architecture.  For practical purposes, with at least 20 Gb of RAM, we find that a size of 15 is optimal.

= A Complete Example =

A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The output files are  ''single.sam''  and  ''pair.1.sam''  and they conform to the .sam format specification.

Karma-colorspace

2009-11-20T02:09:00Z

Tblackw: /* Map Color Space Reads */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfastq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named  "single.sam". 
  

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.1.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

Keep in mind that KARMA requires color space reads that are at least twice as long as the index word size plus two (including the leading primer base).  (For nucleotide space, the minimum read length is twice the word size.)  For example, KARMA uses an index word size of 15 by default, so it will only map color space reads that are 32 base pairs or longer. 

== Auxiliary tools ==

The ABI SOLiD platform generates separate FASTA and quality files named  XXX.csfasta  and  XXX\_QV.qual.  We provide a script  ''solid2csfastq.py''  which converts these into a single color space FASTQ file named  XXX.csfastq.  We believe that a single color space FASTQ file simplifies post processing. 

== Choose an appropriate size for word index ==

The length of the index words influences mapping performance.  Using short index words increases the number of calculation cycles for a single read and duplications of a single word.  On the other side, long index words require much larger memory.  Please also keep in mind that appropriate size is related to your hardware architecture.  For practical purposes, with at least 20 Gb of RAM, we find that a size of 15 is optimal.

= A Complete Example =

A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The output files are  ''single.sam''  and  ''pair.1.sam''  and they conform to the .sam format specification.

Karma-colorspace

2009-11-20T02:07:59Z

Tblackw: /* Map Color Space Reads */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfastq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named  "single.sam". 
  

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.1.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

Keep in mind that KARMA requires color space reads that are at least twice as long as the index word size plus two (including the leading primer base).  (For nucleotide space, the minimum read length is twice the word size.)  For example, KARMA uses an index word size of 15 by default, so it will only map color space reads that are 32 base pairs or longer. 

== Auxiliary tools ==

The ABI SOLiD platform generates separate FASTA and quality files named  XXX.csfasta  and  XXX\_QV.qual.  We provide a script  ''solid2csfastq.py''  which converts these into a single color space FASTQ file named  XXX.csfastq.  We believe that a single color space FASTQ file simplifies post processing. 

== Choose an appropriate size for word index ==

The length of the index words influences mapping performance.  Using short index words increases the number of calculation cycles for a single read and duplications of a single word.  On the other side, long index words require much larger memory.  Please also keep in mind that appropriate size is related to your hardware architecture.  For practical purposes, with at least 20 Gb of RAM, we find that a size of 15 is optimal.

= A Complete Example =

A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The output files are  ''single.sam''  and  ''pair.1.sam''  and they conform to the .sam format specification.

Karma-colorspace

2009-11-20T02:06:48Z

Tblackw: /* A Complete Example */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfastq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named  "single.sam". 
  

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

Keep in mind that KARMA requires color space reads that are at least twice as long as the index word size plus two (including the leading primer base).  (For nucleotide space, the minimum read length is twice the word size.)  For example, KARMA uses an index word size of 15 by default, so it will only map color space reads that are 32 base pairs or longer. 

== Auxiliary tools ==

The ABI SOLiD platform generates separate FASTA and quality files named  XXX.csfasta  and  XXX\_QV.qual.  We provide a script  ''solid2csfastq.py''  which converts these into a single color space FASTQ file named  XXX.csfastq.  We believe that a single color space FASTQ file simplifies post processing. 

== Choose an appropriate size for word index ==

The length of the index words influences mapping performance.  Using short index words increases the number of calculation cycles for a single read and duplications of a single word.  On the other side, long index words require much larger memory.  Please also keep in mind that appropriate size is related to your hardware architecture.  For practical purposes, with at least 20 Gb of RAM, we find that a size of 15 is optimal.

= A Complete Example =

A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The output files are  ''single.sam''  and  ''pair.1.sam''  and they conform to the .sam format specification.

Karma-colorspace

2009-11-20T02:03:46Z

Tblackw: /* Choose an appropriate size for word index */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfastq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named  "single.sam". 
  

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

Keep in mind that KARMA requires color space reads that are at least twice as long as the index word size plus two (including the leading primer base).  (For nucleotide space, the minimum read length is twice the word size.)  For example, KARMA uses an index word size of 15 by default, so it will only map color space reads that are 32 base pairs or longer. 

== Auxiliary tools ==

The ABI SOLiD platform generates separate FASTA and quality files named  XXX.csfasta  and  XXX\_QV.qual.  We provide a script  ''solid2csfastq.py''  which converts these into a single color space FASTQ file named  XXX.csfastq.  We believe that a single color space FASTQ file simplifies post processing. 

== Choose an appropriate size for word index ==

The length of the index words influences mapping performance.  Using short index words increases the number of calculation cycles for a single read and duplications of a single word.  On the other side, long index words require much larger memory.  Please also keep in mind that appropriate size is related to your hardware architecture.  For practical purposes, with at least 20 Gb of RAM, we find that a size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:58:42Z

Tblackw: /* Auxiliary tools */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfastq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named  "single.sam". 
  

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

Keep in mind that KARMA requires color space reads that are at least twice as long as the index word size plus two (including the leading primer base).  (For nucleotide space, the minimum read length is twice the word size.)  For example, KARMA uses an index word size of 15 by default, so it will only map color space reads that are 32 base pairs or longer. 

== Auxiliary tools ==

The ABI SOLiD platform generates separate FASTA and quality files named  XXX.csfasta  and  XXX\_QV.qual.  We provide a script  ''solid2csfastq.py''  which converts these into a single color space FASTQ file named  XXX.csfastq.  We believe that a single color space FASTQ file simplifies post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:57:57Z

Tblackw: /* Auxiliary tools */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfastq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named  "single.sam". 
  

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

Keep in mind that KARMA requires color space reads that are at least twice as long as the index word size plus two (including the leading primer base).  (For nucleotide space, the minimum read length is twice the word size.)  For example, KARMA uses an index word size of 15 by default, so it will only map color space reads that are 32 base pairs or longer. 

== Auxiliary tools ==

The ABI SOLiD platform generates separate FASTA and quality files named  XXX.csfasta  and  XXX\_QV.qual.  We provide a script ''solid2csfastq.py'' which converts these into a single color space FASTQ file named  XXX.csfastq.  We believe that a single color space FASTQ file simplifies post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:57:07Z

Tblackw: /* Auxiliary tools */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfastq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named  "single.sam". 
  

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

Keep in mind that KARMA requires color space reads that are at least twice as long as the index word size plus two (including the leading primer base).  (For nucleotide space, the minimum read length is twice the word size.)  For example, KARMA uses an index word size of 15 by default, so it will only map color space reads that are 32 base pairs or longer. 

== Auxiliary tools ==

The ABI SOLiD platform generates separate FASTA and quality files, usually named  XXX.csfasta  and  XXX\_QV.qual.  We provide a script ''solid2csfastq.py'' which converts these into a single color space FASTQ file named  XXX.csfastq.  We believe that a single color space FASTQ file simplifies post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:54:08Z

Tblackw: /* Auxiliary tools */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfastq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named  "single.sam". 
  

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

Keep in mind that KARMA requires color space reads that are at least twice as long as the index word size plus two (including the leading primer base).  (For nucleotide space, the minimum read length is twice the word size.)  For example, KARMA uses an index word size of 15 by default, so it will only map color space reads that are 32 base pairs or longer. 

== Auxiliary tools ==

The ABI SOLiD platform generates separate FASTA files (e.g. XXX.csfasta) and quality files (e.g. XXX\_QV.qual).  We provide a script, ''solid2csfastq.py'', to convert these into a single color space FASTQ file (e.g. XXX.csfastq).  We believe a single color space FASTQ file simplifies post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:51:06Z

Tblackw: /* Minimum read length requirement */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfastq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named  "single.sam". 
  

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

Keep in mind that KARMA requires color space reads that are at least twice as long as the index word size plus two (including the leading primer base).  (For nucleotide space, the minimum read length is twice the word size.)  For example, KARMA uses an index word size of 15 by default, so it will only map color space reads that are 32 base pairs or longer. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:43:56Z

Tblackw: /* Map Color Space Reads */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfastq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named  "single.sam". 
  

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored in two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a .sam  file named  "pair.sam", which contains reads from both files.  If multiple paired end read files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

  Keep in mind that the requirement of minimum color space read length for KARMA is twice the size of word plus two (including leading primer).   (For nucleotide space, the minimum length requirement is twice the word size.)   For example, KARMA use word size of 15 by default, so it will try to map color space reads that are longer than 32 base pairs. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:38:12Z

Tblackw: /* Map Color Space Reads */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

KARMA expects valid color space FASTQ files as input.  We often use the suffix .csfastq to distinguish these from nucleotide space reads.  For a .csfatq  file of single end color space reads named   single.csfastq,   invoke the command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

This command line specifies both the nucleotide and color space reference sequences (and the word indexes, invisibly).  The output will be written to a file in .sam  format named "single.sam". 

Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single.1.csfastq single.2.csfastq single.3.csfastq

For paired end color space reads, use the option "--pairedReads".  Suppose the paired end reads are stored as two files,  pair.1.csfastq  and  pair.2.csfastq.  The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq

The mapping results will be stored in a SAM file named "pair.sam", which contains reads from both files.  If multiple paired end reads files are specified on the command line, KARMA will pair the 1st and 2nd files, 3rd and 4th files and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair.1.csfastq pair.2.csfastq pair.3.csfastq pair.4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

  Keep in mind that the requirement of minimum color space read length for KARMA is twice the size of word plus two (including leading primer).   (For nucleotide space, the minimum length requirement is twice the word size.)   For example, KARMA use word size of 15 by default, so it will try to map color space reads that are longer than 32 base pairs. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:19:11Z

Tblackw: /* Build Binary Reference Genome and Word Index */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files, one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not exceed half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

  KARMA takes valid color space FASTQ files inputs.   We usually use suffix .csfastq to distinguish it from nucleotide space reads.   For single end color space read, we can invoke command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

  Mapping results are store in a SAM file named "single.sam".     Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single1.csfastq single2.csfastq single3.csfastq

 

  For paired end color space reads, option "--pairedReads" is requires.   Suppose the paired end reads are stored in file, pair1.csfastq and pair2.csfastq.   The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

  Mapping results are store in a SAM file named "pair1.sam", which contains reads from both files.     Similarly multiple paired end reads files can be specified in command line, and KARMA will pair 1st and 2rd file, 3rd and 4th file and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq pair3.csfastq pair4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

  Keep in mind that the requirement of minimum color space read length for KARMA is twice the size of word plus two (including leading primer).   (For nucleotide space, the minimum length requirement is twice the word size.)   For example, KARMA use word size of 15 by default, so it will try to map color space reads that are longer than 32 base pairs. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:17:24Z

Tblackw: /* Build Binary Reference Genome and Word Index */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, one also needs to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not be longer than half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

  KARMA takes valid color space FASTQ files inputs.   We usually use suffix .csfastq to distinguish it from nucleotide space reads.   For single end color space read, we can invoke command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

  Mapping results are store in a SAM file named "single.sam".     Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single1.csfastq single2.csfastq single3.csfastq

 

  For paired end color space reads, option "--pairedReads" is requires.   Suppose the paired end reads are stored in file, pair1.csfastq and pair2.csfastq.   The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

  Mapping results are store in a SAM file named "pair1.sam", which contains reads from both files.     Similarly multiple paired end reads files can be specified in command line, and KARMA will pair 1st and 2rd file, 3rd and 4th file and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq pair3.csfastq pair4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

  Keep in mind that the requirement of minimum color space read length for KARMA is twice the size of word plus two (including leading primer).   (For nucleotide space, the minimum length requirement is twice the word size.)   For example, KARMA use word size of 15 by default, so it will try to map color space reads that are longer than 32 base pairs. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:16:43Z

Tblackw: /* Build Binary Reference Genome and Word Index */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex''  to create both a binary sequence and the word index files.) 

Second, we also need to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is used.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not be longer than half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

  KARMA takes valid color space FASTQ files inputs.   We usually use suffix .csfastq to distinguish it from nucleotide space reads.   For single end color space read, we can invoke command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

  Mapping results are store in a SAM file named "single.sam".     Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single1.csfastq single2.csfastq single3.csfastq

 

  For paired end color space reads, option "--pairedReads" is requires.   Suppose the paired end reads are stored in file, pair1.csfastq and pair2.csfastq.   The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

  Mapping results are store in a SAM file named "pair1.sam", which contains reads from both files.     Similarly multiple paired end reads files can be specified in command line, and KARMA will pair 1st and 2rd file, 3rd and 4th file and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq pair3.csfastq pair4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

  Keep in mind that the requirement of minimum color space read length for KARMA is twice the size of word plus two (including leading primer).   (For nucleotide space, the minimum length requirement is twice the word size.)   For example, KARMA use word size of 15 by default, so it will try to map color space reads that are longer than 32 base pairs. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:14:42Z

Tblackw: /* Build Binary Reference Genome and Word Index */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference).  Suppose that   NCBI36.fa   is a FASTA file which contains nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex'' to create both a binary sequence and the word index files.) 

Second, we also need to build color space versions of both the genome reference sequence (option: --createReference) and the word index files (option: --createIndex).  The same nucleotide FASTA file is needed.  However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity.  The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
When building the index files one can set the word length for indexing.  We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM.  Shorter index words will decrease the memory footprint at the cost of increased run time.  However, the word length must not be longer than half the length of the color space reads you intend to map, minus 1.  (See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion.)  Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

  KARMA takes valid color space FASTQ files inputs.   We usually use suffix .csfastq to distinguish it from nucleotide space reads.   For single end color space read, we can invoke command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

  Mapping results are store in a SAM file named "single.sam".     Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single1.csfastq single2.csfastq single3.csfastq

 

  For paired end color space reads, option "--pairedReads" is requires.   Suppose the paired end reads are stored in file, pair1.csfastq and pair2.csfastq.   The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

  Mapping results are store in a SAM file named "pair1.sam", which contains reads from both files.     Similarly multiple paired end reads files can be specified in command line, and KARMA will pair 1st and 2rd file, 3rd and 4th file and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq pair3.csfastq pair4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

  Keep in mind that the requirement of minimum color space read length for KARMA is twice the size of word plus two (including leading primer).   (For nucleotide space, the minimum length requirement is twice the word size.)   For example, KARMA use word size of 15 by default, so it will try to map color space reads that are longer than 32 base pairs. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T01:05:07Z

Tblackw: /* Build Binary Reference Genome and Word Index */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

First, build a binary version of the genome reference sequence as nucleotides (option: --createReference). Suppose that NCBI36.fa is a FASTA file which contains the nucleotide sequences for all chromosomes. 
The command to invoke is: 

karma --createReference --reference NCBI36.fa
 
(To let KARMA map nucleotide space reads, one would use instead ''--createIndex'' to create both a binary sequence and the word index files.) 

Second, we also need to build a binary version of the genome reference sequence (option: --createReference) and the word index files (option: --createIndex) in color space. The same nucleotide FASTA file is needed. However, to avoid naming conflicts among the resulting binary files, we suggest appending "CS" to the base file name for clarity. The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa
 
An important parameter is the word length for indexing. We recommend N = 15 (the default value) for the human genome on a machine with at least 20 Gb of RAM. Shorter words will decrease the memory footprint at the cost of increased run time. However, the word length must not be longer than half the length of the color space reads you intend to map, minus 1. See [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion. Specify ``--wordSize N`` in order to use ''N'' as the word size. 

 

= Map Color Space Reads =

  KARMA takes valid color space FASTQ files inputs.   We usually use suffix .csfastq to distinguish it from nucleotide space reads.   For single end color space read, we can invoke command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

  Mapping results are store in a SAM file named "single.sam".     Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single1.csfastq single2.csfastq single3.csfastq

 

  For paired end color space reads, option "--pairedReads" is requires.   Suppose the paired end reads are stored in file, pair1.csfastq and pair2.csfastq.   The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

  Mapping results are store in a SAM file named "pair1.sam", which contains reads from both files.     Similarly multiple paired end reads files can be specified in command line, and KARMA will pair 1st and 2rd file, 3rd and 4th file and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq pair3.csfastq pair4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

  Keep in mind that the requirement of minimum color space read length for KARMA is twice the size of word plus two (including leading primer).   (For nucleotide space, the minimum length requirement is twice the word size.)   For example, KARMA use word size of 15 by default, so it will try to map color space reads that are longer than 32 base pairs. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T00:33:11Z

Tblackw: /* Overview */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at a speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize the input data requirements as following:

*  A binary conversion of the genome reference sequence as nucleotides (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]]})
*  A binary conversion of the genome reference sequence as colors plus word indices in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for a description)
*  Color space reads longer than a minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example demonstrating the whole procedure from building the word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

  First, we need to build binary reference genome (option: --createReference)   (To let KARMA map nucleotide space reads, you need to use ''--createIndex'' to create the word index file.) 

  in nucleotide space. Assume NCBI36.fa is a FASTA file contains sequences of all chromosomes.   The command to invoke is: 

karma --createReference --reference NCBI36.fa

 

  Second, we need to build binary reference genome (option: --createReference) and word index (option: --createIndex)   in color space. The same FASTA file is needed. However, to avoid naming conflicts, we suggest using word "CS"   appending to the base file name for clarity. The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

  An important parameter is the size of words for indexing.   We recommand 15 (default value) for human reference genome.   Specifiy ``--wordSize N`` if you like to use ''N'' as word size.   Typically you will observe performance change (see [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion).       Note, multiple chromosomes are supported.   In current version, KARMA can take one FASTA file which contains sequences of all chromosomes. 

 

= Map Color Space Reads =

  KARMA takes valid color space FASTQ files inputs.   We usually use suffix .csfastq to distinguish it from nucleotide space reads.   For single end color space read, we can invoke command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

  Mapping results are store in a SAM file named "single.sam".     Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single1.csfastq single2.csfastq single3.csfastq

 

  For paired end color space reads, option "--pairedReads" is requires.   Suppose the paired end reads are stored in file, pair1.csfastq and pair2.csfastq.   The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

  Mapping results are store in a SAM file named "pair1.sam", which contains reads from both files.     Similarly multiple paired end reads files can be specified in command line, and KARMA will pair 1st and 2rd file, 3rd and 4th file and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq pair3.csfastq pair4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

  Keep in mind that the requirement of minimum color space read length for KARMA is twice the size of word plus two (including leading primer).   (For nucleotide space, the minimum length requirement is twice the word size.)   For example, KARMA use word size of 15 by default, so it will try to map color space reads that are longer than 32 base pairs. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T00:14:42Z

Tblackw: /* Overview */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at the speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize software requirements as following:

*  Binary reference genome in nucleotide space (see [[#Input_file_requirement|Input file requirement]]})
*  Binary reference genome and word index in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in valid color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for file specification)
*  Color space reads are longer than minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine without using more memory than running a single instance. 
*30G disk space

 We show a complete example reviewing the whole procedure from building word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

  First, we need to build binary reference genome (option: --createReference)   (To let KARMA map nucleotide space reads, you need to use ''--createIndex'' to create the word index file.) 

  in nucleotide space. Assume NCBI36.fa is a FASTA file contains sequences of all chromosomes.   The command to invoke is: 

karma --createReference --reference NCBI36.fa

 

  Second, we need to build binary reference genome (option: --createReference) and word index (option: --createIndex)   in color space. The same FASTA file is needed. However, to avoid naming conflicts, we suggest using word "CS"   appending to the base file name for clarity. The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

  An important parameter is the size of words for indexing.   We recommand 15 (default value) for human reference genome.   Specifiy ``--wordSize N`` if you like to use ''N'' as word size.   Typically you will observe performance change (see [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion).       Note, multiple chromosomes are supported.   In current version, KARMA can take one FASTA file which contains sequences of all chromosomes. 

 

= Map Color Space Reads =

  KARMA takes valid color space FASTQ files inputs.   We usually use suffix .csfastq to distinguish it from nucleotide space reads.   For single end color space read, we can invoke command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

  Mapping results are store in a SAM file named "single.sam".     Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single1.csfastq single2.csfastq single3.csfastq

 

  For paired end color space reads, option "--pairedReads" is requires.   Suppose the paired end reads are stored in file, pair1.csfastq and pair2.csfastq.   The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

  Mapping results are store in a SAM file named "pair1.sam", which contains reads from both files.     Similarly multiple paired end reads files can be specified in command line, and KARMA will pair 1st and 2rd file, 3rd and 4th file and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq pair3.csfastq pair4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

  Keep in mind that the requirement of minimum color space read length for KARMA is twice the size of word plus two (including leading primer).   (For nucleotide space, the minimum length requirement is twice the word size.)   For example, KARMA use word size of 15 by default, so it will try to map color space reads that are longer than 32 base pairs. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.

Karma-colorspace

2009-11-20T00:13:14Z

Tblackw: /* Overview */

= Overview =

KARMA (K-tuple Alignment with Rapid Matching Algorithm) is able to map 35 bp single end color space reads at the speed of approximately <math>1.2-2.0 \times 10^9</math> reads per hour using Intel Xeon X760 2.66GHz and 128G memory.

We summarize software requirements as following:

*  Binary reference genome in nucleotide space (see [[#Input_file_requirement|Input file requirement]]})
*  Binary reference genome and word index in color space (see [[#Build_Binary_Reference_Genome_and_Word_Index|Build Binary Reference Genome and Word Index]])
*  Color space reads in valid color space FASTQ format (see [[#Input_file_requirement|Input file requirement]] for file specification)
*  Color space reads are longer than minimum length requirement. (see [[#Minimum_read_length_requirement|Minimum read length requirement]])
*  Specify color space parameter when starting KARMA (see [[#Map_Color_Space_Reads|Map Color Space Reads]]) 

Please note the hardware requirements for KARMA are:

*20G memory. By using shared memory for the word index tables, multiple instances of KARMA can run on one machine yet consume about the same amount of memory as running one process. 
*30G disk space

 We show a complete example reviewing the whole procedure from building word index to mapping color space reads in [[#A_Complete_Example|A Complete Example]].

= Build Binary Reference Genome and Word Index =

  First, we need to build binary reference genome (option: --createReference)   (To let KARMA map nucleotide space reads, you need to use ''--createIndex'' to create the word index file.) 

  in nucleotide space. Assume NCBI36.fa is a FASTA file contains sequences of all chromosomes.   The command to invoke is: 

karma --createReference --reference NCBI36.fa

 

  Second, we need to build binary reference genome (option: --createReference) and word index (option: --createIndex)   in color space. The same FASTA file is needed. However, to avoid naming conflicts, we suggest using word "CS"   appending to the base file name for clarity. The command to invoke is: 

ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

  An important parameter is the size of words for indexing.   We recommand 15 (default value) for human reference genome.   Specifiy ``--wordSize N`` if you like to use ''N'' as word size.   Typically you will observe performance change (see [[#Choose_an_appropriate_size_for_word_index|Choose an appropriate size for word index]] for more discussion).       Note, multiple chromosomes are supported.   In current version, KARMA can take one FASTA file which contains sequences of all chromosomes. 

 

= Map Color Space Reads =

  KARMA takes valid color space FASTQ files inputs.   We usually use suffix .csfastq to distinguish it from nucleotide space reads.   For single end color space read, we can invoke command: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq

  Mapping results are store in a SAM file named "single.sam".     Multiple input files are also acceptable, e.g. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
single1.csfastq single2.csfastq single3.csfastq

 

  For paired end color space reads, option "--pairedReads" is requires.   Suppose the paired end reads are stored in file, pair1.csfastq and pair2.csfastq.   The command to invoke is: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

  Mapping results are store in a SAM file named "pair1.sam", which contains reads from both files.     Similarly multiple paired end reads files can be specified in command line, and KARMA will pair 1st and 2rd file, 3rd and 4th file and etc. 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq pair3.csfastq pair4.csfastq

= Additional Information =

== Input file requirement ==

KARMA requires input files in color space FASTQ format. The length of each read (which includes the leading primer base) should equal the length of its quality string. An example of a valid color space FASTQ file follows:

@Chromosome_20_048435095_Genome_2757096147
A02232200222021320012102212311002212
+
!!1111111111111111111111111111111111

== Minimum read length requirement ==

  Keep in mind that the requirement of minimum color space read length for KARMA is twice the size of word plus two (including leading primer).   (For nucleotide space, the minimum length requirement is twice the word size.)   For example, KARMA use word size of 15 by default, so it will try to map color space reads that are longer than 32 base pairs. 

== Auxiliary tools ==

  ABI SOLiD platform generated FASTA file (e.g. XXX.csfasta) and quality file (e.g. XXX\_QV.qual) separately. We wrote a script, ''solid2csfastq.py'', to convert it to color space FASTQ file(e.g. XXX.csfastq). We believe a single color space FASTQ file will simplify post processing. 

== Choose an appropriate size for word index ==

Size for word index is sensitive to mapping performance. A small size of word index will increase the number of calculation cycles for a single read and duplications of a single word. On the other side, a big size will require much larger memory. Please also keep in mind that appropriate size is related to your hardware architecture. For practically purpose, we found size of 15 is optimal.

= A Complete Example =

  A wrap-up message for quick start mapping color space reads. 

Building binary genome reference and word index: 

karma --createReference --reference NCBI36.fa
ln -s NCBI36.fa NCBI36CS.fa
karma --colorSpace --createReference --createIndex --reference NCBI36CS.fa

 

Mapping color space reads: 

karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace single.csfastq
karma --reference NCBI36.fa --csReference NCBI36CS.fa --colorSpace \
--pairedReads pair1.csfastq pair2.csfastq

The output files are ''single.sam'' and ''pair1.sam'' and they conform SAM specification.